Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 51 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......XXXXXX..] duration: 5.641289151s FAIL (2025-07-27 00:56:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== false Jul 27 00:54:57.821101936 (61h19m7.543664977s) [upload=58793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:57.821108168 (61h19m7.543671208s) [upload=58793]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:58.268999381 (61h19m7.991562411s) [seg_pos={0 0}, upload=58793]: Begin upload segment... Jul 27 00:54:58.269047784 (61h19m7.991610824s) [seg_pos={0 0}, upload=58793]: Done begin upload segment. Jul 27 00:54:58.269068714 (61h19m7.991631754s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269108290 (61h19m7.99167133s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269416039 (61h19m7.991979079s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269588800 (61h19m7.99215184s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269666007 (61h19m7.992229048s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269938439 (61h19m7.992501479s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270186764 (61h19m7.992749794s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270456489 (61h19m7.993019529s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270693122 (61h19m7.993256162s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270924895 (61h19m7.993487935s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270973609 (61h19m7.993536649s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271180806 (61h19m7.993743846s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271367583 (61h19m7.993930613s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271609946 (61h19m7.994172986s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271884882 (61h19m7.994447922s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271925130 (61h19m7.99448817s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272230464 (61h19m7.994793504s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272352857 (61h19m7.994915898s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:59.403085511 (61h19m9.125648551s) [upload=58793]: Next returned nil segment Jul 27 00:54:59.403106221 (61h19m9.125669281s) [upload=58793]: Waiting for error group managing segments... Jul 27 00:54:59.699317478 (61h19m9.421880518s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.712155049 (61h19m9.434718079s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.718874015 (61h19m9.441437055s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.219903397 (61h19m9.942466437s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.393745135 (61h19m10.116308165s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.589079921 (61h19m11.311642972s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.684825540 (61h19m11.40738858s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:02.296389006 (61h19m12.018952046s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.015602295 (61h19m12.738165335s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.588380603 (61h19m13.310943643s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:07.419707501 (61h19m17.142270581s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:10.064731586 (61h19m19.787294626s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:57.170253087 (61h20m6.892816127s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1fPp4DvLe18dZWPmMPFiC8o9jL4uiEFNmPAHHmFEH74ZXNndw3, address:104.36.82.53:4153): protocol: write tcp 188.166.225.170:41520->104.36.82.53:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170329522 (61h20m6.892892562s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170370181 (61h20m6.892933221s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:17wSQzrFc2hUimSsNwDxCgDDi4TpHkbCEH38fGKf9B8WqWzxpW, address:104.36.82.52:4091): protocol: rpc: dial tcp 104.36.82.52:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170401270 (61h20m6.89296431s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: rpc: dial tcp 104.36.82.46:4500: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170514156 (61h20m6.893077196s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1AivThSVLAp9zneNK79sic4AqnTQ62yvNTFF36imigarMGqDSW, address:104.36.82.49:4005): protocol: rpc: dial tcp 104.36.82.49:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170524104 (61h20m6.893087155s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:126WPF6MtX7YjuKf73rGJrifdXyd78KY4FdL757qDipGNHBW6uY, address:104.36.82.15:4289): protocol: rpc: dial tcp 104.36.82.15:4289: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170556958 (61h20m6.893119998s) [seg_pos={0 0}, upload=58793]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:55:57.400256958 (61h20m7.122819998s) [upload=58793]: Done uploading segments... Jul 27 00:55:57.400281876 (61h20m7.122844916s) [upload=58793]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.641307455s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 35 out of 60 is failed (threshold=3) [X.XX.X.XXXXXXXXXX.X.XX..XXXXX.XXXX..XXXX.....X....X....XXX..] duration: 5.739802978s FAIL (2025-07-27 00:56:57 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.739820712s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......XXXXXX.] duration: 54.358274097s FAIL (2025-07-27 00:56:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== false Jul 27 00:54:57.821101936 (61h19m7.543664977s) [upload=58793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:57.821108168 (61h19m7.543671208s) [upload=58793]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:58.268999381 (61h19m7.991562411s) [seg_pos={0 0}, upload=58793]: Begin upload segment... Jul 27 00:54:58.269047784 (61h19m7.991610824s) [seg_pos={0 0}, upload=58793]: Done begin upload segment. Jul 27 00:54:58.269068714 (61h19m7.991631754s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269108290 (61h19m7.99167133s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269416039 (61h19m7.991979079s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269588800 (61h19m7.99215184s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269666007 (61h19m7.992229048s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269938439 (61h19m7.992501479s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270186764 (61h19m7.992749794s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270456489 (61h19m7.993019529s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270693122 (61h19m7.993256162s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270924895 (61h19m7.993487935s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270973609 (61h19m7.993536649s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271180806 (61h19m7.993743846s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271367583 (61h19m7.993930613s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271609946 (61h19m7.994172986s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271884882 (61h19m7.994447922s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271925130 (61h19m7.99448817s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272230464 (61h19m7.994793504s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272352857 (61h19m7.994915898s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:59.403085511 (61h19m9.125648551s) [upload=58793]: Next returned nil segment Jul 27 00:54:59.403106221 (61h19m9.125669281s) [upload=58793]: Waiting for error group managing segments... Jul 27 00:54:59.699317478 (61h19m9.421880518s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.712155049 (61h19m9.434718079s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.718874015 (61h19m9.441437055s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.219903397 (61h19m9.942466437s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.393745135 (61h19m10.116308165s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.589079921 (61h19m11.311642972s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.684825540 (61h19m11.40738858s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:02.296389006 (61h19m12.018952046s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.015602295 (61h19m12.738165335s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.588380603 (61h19m13.310943643s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:07.419707501 (61h19m17.142270581s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:10.064731586 (61h19m19.787294626s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:57.170253087 (61h20m6.892816127s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1fPp4DvLe18dZWPmMPFiC8o9jL4uiEFNmPAHHmFEH74ZXNndw3, address:104.36.82.53:4153): protocol: write tcp 188.166.225.170:41520->104.36.82.53:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170329522 (61h20m6.892892562s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170370181 (61h20m6.892933221s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:17wSQzrFc2hUimSsNwDxCgDDi4TpHkbCEH38fGKf9B8WqWzxpW, address:104.36.82.52:4091): protocol: rpc: dial tcp 104.36.82.52:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170401270 (61h20m6.89296431s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: rpc: dial tcp 104.36.82.46:4500: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170514156 (61h20m6.893077196s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1AivThSVLAp9zneNK79sic4AqnTQ62yvNTFF36imigarMGqDSW, address:104.36.82.49:4005): protocol: rpc: dial tcp 104.36.82.49:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170524104 (61h20m6.893087155s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:126WPF6MtX7YjuKf73rGJrifdXyd78KY4FdL757qDipGNHBW6uY, address:104.36.82.15:4289): protocol: rpc: dial tcp 104.36.82.15:4289: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170556958 (61h20m6.893119998s) [seg_pos={0 0}, upload=58793]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:55:57.400256958 (61h20m7.122819998s) [upload=58793]: Done uploading segments... Jul 27 00:55:57.400281876 (61h20m7.122844916s) [upload=58793]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 54.358295087s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [X..XX....XX..XXXXX.X.XXXXXXXXX.XXXXXXXX..X.X.X..........X...] duration: 6.065960719s FAIL (2025-07-27 00:56:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7326QE72JPEXPXFHPPYHSNKIURIMRGIQ, seg_pos={0 0}, upload=58778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:33562->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:03.385284649 (61h18m13.107847679s) [seg_pos={0 0}, upload=58778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:54:03.385367337 (61h18m13.107930377s) [upload=58778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:54:03.385371374 (61h18m13.107934414s) [upload=58778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:54:03.688084400 (61h18m13.41064744s) [upload=58778]: Done uploading segments... Jul 27 00:54:03.688101954 (61h18m13.410664994s) [upload=58778]: Upload finished. err: <nil> Jul 27 00:54:29.969393323 (61h18m39.691956364s) [upload=58786]: Starting upload Jul 27 00:54:29.969484158 (61h18m39.692047198s) [upload=58786]: Done starting upload Jul 27 00:54:29.969609477 (61h18m39.692172517s) [upload=58786]: Uploading segments... Jul 27 00:54:29.969791546 (61h18m39.692354576s) [upload=58786]: Got next segment. Inline: false Jul 27 00:54:29.969897859 (61h18m39.6924609s) [upload=58786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:29.969903801 (61h18m39.692466841s) [upload=58786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:30.202493651 (61h18m39.925056691s) [seg_pos={0 0}, upload=58786]: Begin upload segment... Jul 27 00:54:30.202544298 (61h18m39.925107338s) [seg_pos={0 0}, upload=58786]: Done begin upload segment. Jul 27 00:54:30.202565960 (61h18m39.92512899s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202629651 (61h18m39.925192691s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202926890 (61h18m39.92548992s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203102106 (61h18m39.925665146s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203204351 (61h18m39.925767391s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203445583 (61h18m39.926008623s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203491891 (61h18m39.926054921s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203718445 (61h18m39.926281475s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204010243 (61h18m39.926573283s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204257396 (61h18m39.926820436s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204507925 (61h18m39.927070976s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205281365 (61h18m39.927844435s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205688314 (61h18m39.928251354s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205981905 (61h18m39.928544945s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.206218718 (61h18m39.928781759s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.207082942 (61h18m39.929645982s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:31.892857589 (61h18m41.615420639s) [upload=58786]: Next returned nil segment Jul 27 00:54:31.892877968 (61h18m41.615441008s) [upload=58786]: Waiting for error group managing segments... Jul 27 00:54:32.223482398 (61h18m41.946045438s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.237060637 (61h18m41.959623677s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.299221539 (61h18m42.021784619s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325716521 (61h18m42.048279561s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325884473 (61h18m42.048447503s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.372983466 (61h18m42.095546506s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.379844103 (61h18m42.102407133s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.384666850 (61h18m42.10722989s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.472371113 (61h18m42.194934143s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:33.667049984 (61h18m43.389613024s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:34.213712174 (61h18m43.936275214s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:42.252927986 (61h18m51.975491026s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:59.328639826 (61h19m9.051202866s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328713146 (61h19m9.051276186s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328897458 (61h19m9.051460488s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 188.166.225.170:42398->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329496955 (61h19m9.052059995s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329528055 (61h19m9.052091095s) [seg_pos={0 0}, upload=58786]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:54:59.557689493 (61h19m9.280252533s) [upload=58786]: Done uploading segments... Jul 27 00:54:59.557710713 (61h19m9.280273753s) [upload=58786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.065984846s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [XXX.....X..........X..X..X...XX.............................] duration: 8.098323439s FAIL (2025-07-27 00:56:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.098346984s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 36 out of 60 is failed (threshold=3) [XX.XX.X.XXXXXXXXXX.X.XX..XXXXX.XXXX..XXXX.....X....X....XXX.] duration: 7.457041033s FAIL (2025-07-27 00:56:35 UTC) orbiter warning: uplink: context deadline exceeded Duration: 7.457064598s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [...........XX......X...XXX..XXX..X..X.X..............X......] duration: 22.241599127s FAIL (2025-07-27 00:56:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.241624787s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [.X..XX....XX..XXXXX.X.XXXXXXXXX.XXXXXXXX..X.X.X..........X..] duration: 5.73559578s FAIL (2025-07-27 00:56:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7326QE72JPEXPXFHPPYHSNKIURIMRGIQ, seg_pos={0 0}, upload=58778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:33562->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:03.385284649 (61h18m13.107847679s) [seg_pos={0 0}, upload=58778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:54:03.385367337 (61h18m13.107930377s) [upload=58778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:54:03.385371374 (61h18m13.107934414s) [upload=58778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:54:03.688084400 (61h18m13.41064744s) [upload=58778]: Done uploading segments... Jul 27 00:54:03.688101954 (61h18m13.410664994s) [upload=58778]: Upload finished. err: <nil> Jul 27 00:54:29.969393323 (61h18m39.691956364s) [upload=58786]: Starting upload Jul 27 00:54:29.969484158 (61h18m39.692047198s) [upload=58786]: Done starting upload Jul 27 00:54:29.969609477 (61h18m39.692172517s) [upload=58786]: Uploading segments... Jul 27 00:54:29.969791546 (61h18m39.692354576s) [upload=58786]: Got next segment. Inline: false Jul 27 00:54:29.969897859 (61h18m39.6924609s) [upload=58786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:29.969903801 (61h18m39.692466841s) [upload=58786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:30.202493651 (61h18m39.925056691s) [seg_pos={0 0}, upload=58786]: Begin upload segment... Jul 27 00:54:30.202544298 (61h18m39.925107338s) [seg_pos={0 0}, upload=58786]: Done begin upload segment. Jul 27 00:54:30.202565960 (61h18m39.92512899s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202629651 (61h18m39.925192691s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202926890 (61h18m39.92548992s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203102106 (61h18m39.925665146s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203204351 (61h18m39.925767391s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203445583 (61h18m39.926008623s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203491891 (61h18m39.926054921s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203718445 (61h18m39.926281475s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204010243 (61h18m39.926573283s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204257396 (61h18m39.926820436s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204507925 (61h18m39.927070976s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205281365 (61h18m39.927844435s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205688314 (61h18m39.928251354s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205981905 (61h18m39.928544945s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.206218718 (61h18m39.928781759s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.207082942 (61h18m39.929645982s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:31.892857589 (61h18m41.615420639s) [upload=58786]: Next returned nil segment Jul 27 00:54:31.892877968 (61h18m41.615441008s) [upload=58786]: Waiting for error group managing segments... Jul 27 00:54:32.223482398 (61h18m41.946045438s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.237060637 (61h18m41.959623677s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.299221539 (61h18m42.021784619s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325716521 (61h18m42.048279561s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325884473 (61h18m42.048447503s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.372983466 (61h18m42.095546506s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.379844103 (61h18m42.102407133s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.384666850 (61h18m42.10722989s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.472371113 (61h18m42.194934143s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:33.667049984 (61h18m43.389613024s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:34.213712174 (61h18m43.936275214s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:42.252927986 (61h18m51.975491026s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:59.328639826 (61h19m9.051202866s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328713146 (61h19m9.051276186s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328897458 (61h19m9.051460488s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 188.166.225.170:42398->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329496955 (61h19m9.052059995s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329528055 (61h19m9.052091095s) [seg_pos={0 0}, upload=58786]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:54:59.557689493 (61h19m9.280252533s) [upload=58786]: Done uploading segments... Jul 27 00:54:59.557710713 (61h19m9.280273753s) [upload=58786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.735618453s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 36 out of 60 is failed (threshold=3) [.XX.XX.X.XXXXXXXXXX.X.XX..XXXXX.XXXX..XXXX.....X....X....XXX] duration: 30.00213924s FAIL (2025-07-27 00:56:06 UTC) orbiter warning: uplink: context deadline exceeded Duration: 30.002154849s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [............XX......X...XXX..XXX..X..X.X..............X.....] duration: 32.982361796s FAIL (2025-07-27 00:56:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 32.98279314s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 53 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......XXXXXX] duration: 1m0.232014339s FAIL (2025-07-27 00:55:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== false Jul 27 00:54:57.821101936 (61h19m7.543664977s) [upload=58793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:57.821108168 (61h19m7.543671208s) [upload=58793]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:58.268999381 (61h19m7.991562411s) [seg_pos={0 0}, upload=58793]: Begin upload segment... Jul 27 00:54:58.269047784 (61h19m7.991610824s) [seg_pos={0 0}, upload=58793]: Done begin upload segment. Jul 27 00:54:58.269068714 (61h19m7.991631754s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269108290 (61h19m7.99167133s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269416039 (61h19m7.991979079s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269588800 (61h19m7.99215184s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269666007 (61h19m7.992229048s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269938439 (61h19m7.992501479s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270186764 (61h19m7.992749794s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270456489 (61h19m7.993019529s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270693122 (61h19m7.993256162s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270924895 (61h19m7.993487935s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270973609 (61h19m7.993536649s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271180806 (61h19m7.993743846s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271367583 (61h19m7.993930613s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271609946 (61h19m7.994172986s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271884882 (61h19m7.994447922s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271925130 (61h19m7.99448817s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272230464 (61h19m7.994793504s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272352857 (61h19m7.994915898s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:59.403085511 (61h19m9.125648551s) [upload=58793]: Next returned nil segment Jul 27 00:54:59.403106221 (61h19m9.125669281s) [upload=58793]: Waiting for error group managing segments... Jul 27 00:54:59.699317478 (61h19m9.421880518s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.712155049 (61h19m9.434718079s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.718874015 (61h19m9.441437055s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.219903397 (61h19m9.942466437s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.393745135 (61h19m10.116308165s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.589079921 (61h19m11.311642972s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.684825540 (61h19m11.40738858s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:02.296389006 (61h19m12.018952046s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.015602295 (61h19m12.738165335s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.588380603 (61h19m13.310943643s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:07.419707501 (61h19m17.142270581s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:10.064731586 (61h19m19.787294626s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:57.170253087 (61h20m6.892816127s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1fPp4DvLe18dZWPmMPFiC8o9jL4uiEFNmPAHHmFEH74ZXNndw3, address:104.36.82.53:4153): protocol: write tcp 188.166.225.170:41520->104.36.82.53:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170329522 (61h20m6.892892562s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170370181 (61h20m6.892933221s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:17wSQzrFc2hUimSsNwDxCgDDi4TpHkbCEH38fGKf9B8WqWzxpW, address:104.36.82.52:4091): protocol: rpc: dial tcp 104.36.82.52:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170401270 (61h20m6.89296431s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: rpc: dial tcp 104.36.82.46:4500: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170514156 (61h20m6.893077196s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1AivThSVLAp9zneNK79sic4AqnTQ62yvNTFF36imigarMGqDSW, address:104.36.82.49:4005): protocol: rpc: dial tcp 104.36.82.49:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170524104 (61h20m6.893087155s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:126WPF6MtX7YjuKf73rGJrifdXyd78KY4FdL757qDipGNHBW6uY, address:104.36.82.15:4289): protocol: rpc: dial tcp 104.36.82.15:4289: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170556958 (61h20m6.893119998s) [seg_pos={0 0}, upload=58793]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:55:57.400256958 (61h20m7.122819998s) [upload=58793]: Done uploading segments... Jul 27 00:55:57.400281876 (61h20m7.122844916s) [upload=58793]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.232028215s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....................X.................................X....] duration: 1.80484898s FAIL (2025-07-27 00:55:48 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.804876585s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [XXXX.....X..........X..X..X...XX............................] duration: 6.231449779s FAIL (2025-07-27 00:55:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.231495026s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [..X..XX....XX..XXXXX.X.XXXXXXXXX.XXXXXXXX..X.X.X..........X.] duration: 5.849572837s FAIL (2025-07-27 00:55:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7326QE72JPEXPXFHPPYHSNKIURIMRGIQ, seg_pos={0 0}, upload=58778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:33562->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:03.385284649 (61h18m13.107847679s) [seg_pos={0 0}, upload=58778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:54:03.385367337 (61h18m13.107930377s) [upload=58778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:54:03.385371374 (61h18m13.107934414s) [upload=58778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:54:03.688084400 (61h18m13.41064744s) [upload=58778]: Done uploading segments... Jul 27 00:54:03.688101954 (61h18m13.410664994s) [upload=58778]: Upload finished. err: <nil> Jul 27 00:54:29.969393323 (61h18m39.691956364s) [upload=58786]: Starting upload Jul 27 00:54:29.969484158 (61h18m39.692047198s) [upload=58786]: Done starting upload Jul 27 00:54:29.969609477 (61h18m39.692172517s) [upload=58786]: Uploading segments... Jul 27 00:54:29.969791546 (61h18m39.692354576s) [upload=58786]: Got next segment. Inline: false Jul 27 00:54:29.969897859 (61h18m39.6924609s) [upload=58786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:29.969903801 (61h18m39.692466841s) [upload=58786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:30.202493651 (61h18m39.925056691s) [seg_pos={0 0}, upload=58786]: Begin upload segment... Jul 27 00:54:30.202544298 (61h18m39.925107338s) [seg_pos={0 0}, upload=58786]: Done begin upload segment. Jul 27 00:54:30.202565960 (61h18m39.92512899s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202629651 (61h18m39.925192691s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202926890 (61h18m39.92548992s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203102106 (61h18m39.925665146s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203204351 (61h18m39.925767391s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203445583 (61h18m39.926008623s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203491891 (61h18m39.926054921s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203718445 (61h18m39.926281475s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204010243 (61h18m39.926573283s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204257396 (61h18m39.926820436s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204507925 (61h18m39.927070976s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205281365 (61h18m39.927844435s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205688314 (61h18m39.928251354s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205981905 (61h18m39.928544945s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.206218718 (61h18m39.928781759s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.207082942 (61h18m39.929645982s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:31.892857589 (61h18m41.615420639s) [upload=58786]: Next returned nil segment Jul 27 00:54:31.892877968 (61h18m41.615441008s) [upload=58786]: Waiting for error group managing segments... Jul 27 00:54:32.223482398 (61h18m41.946045438s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.237060637 (61h18m41.959623677s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.299221539 (61h18m42.021784619s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325716521 (61h18m42.048279561s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325884473 (61h18m42.048447503s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.372983466 (61h18m42.095546506s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.379844103 (61h18m42.102407133s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.384666850 (61h18m42.10722989s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.472371113 (61h18m42.194934143s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:33.667049984 (61h18m43.389613024s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:34.213712174 (61h18m43.936275214s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:42.252927986 (61h18m51.975491026s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:59.328639826 (61h19m9.051202866s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328713146 (61h19m9.051276186s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328897458 (61h19m9.051460488s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 188.166.225.170:42398->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329496955 (61h19m9.052059995s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329528055 (61h19m9.052091095s) [seg_pos={0 0}, upload=58786]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:54:59.557689493 (61h19m9.280252533s) [upload=58786]: Done uploading segments... Jul 27 00:54:59.557710713 (61h19m9.280273753s) [upload=58786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.849608477s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.XXXX.....X..........X..X..X...XX...........................] duration: 7.534102253s FAIL (2025-07-27 00:55:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.534123944s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 35 out of 60 is failed (threshold=3) [..XX.XX.X.XXXXXXXXXX.X.XX..XXXXX.XXXX..XXXX.....X....X....XX] duration: 5.318838272s FAIL (2025-07-27 00:55:06 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/4M Duration: 5.318854391s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 34 out of 60 is failed (threshold=3) [...XX.XX.X.XXXXXXXXXX.X.XX..XXXXX.XXXX..XXXX.....X....X....X] duration: 30.236992653s FAIL (2025-07-27 00:55:00 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.23701171s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [X..X..XX....XX..XXXXX.X.XXXXXXXXX.XXXXXXXX..X.X.X..........X] duration: 30.230702225s FAIL (2025-07-27 00:54:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7326QE72JPEXPXFHPPYHSNKIURIMRGIQ, seg_pos={0 0}, upload=58778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:33562->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:03.385284649 (61h18m13.107847679s) [seg_pos={0 0}, upload=58778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:54:03.385367337 (61h18m13.107930377s) [upload=58778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:54:03.385371374 (61h18m13.107934414s) [upload=58778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:54:03.688084400 (61h18m13.41064744s) [upload=58778]: Done uploading segments... Jul 27 00:54:03.688101954 (61h18m13.410664994s) [upload=58778]: Upload finished. err: <nil> Jul 27 00:54:29.969393323 (61h18m39.691956364s) [upload=58786]: Starting upload Jul 27 00:54:29.969484158 (61h18m39.692047198s) [upload=58786]: Done starting upload Jul 27 00:54:29.969609477 (61h18m39.692172517s) [upload=58786]: Uploading segments... Jul 27 00:54:29.969791546 (61h18m39.692354576s) [upload=58786]: Got next segment. Inline: false Jul 27 00:54:29.969897859 (61h18m39.6924609s) [upload=58786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:29.969903801 (61h18m39.692466841s) [upload=58786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:30.202493651 (61h18m39.925056691s) [seg_pos={0 0}, upload=58786]: Begin upload segment... Jul 27 00:54:30.202544298 (61h18m39.925107338s) [seg_pos={0 0}, upload=58786]: Done begin upload segment. Jul 27 00:54:30.202565960 (61h18m39.92512899s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202629651 (61h18m39.925192691s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202926890 (61h18m39.92548992s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203102106 (61h18m39.925665146s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203204351 (61h18m39.925767391s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203445583 (61h18m39.926008623s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203491891 (61h18m39.926054921s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203718445 (61h18m39.926281475s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204010243 (61h18m39.926573283s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204257396 (61h18m39.926820436s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204507925 (61h18m39.927070976s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205281365 (61h18m39.927844435s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205688314 (61h18m39.928251354s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205981905 (61h18m39.928544945s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.206218718 (61h18m39.928781759s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.207082942 (61h18m39.929645982s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:31.892857589 (61h18m41.615420639s) [upload=58786]: Next returned nil segment Jul 27 00:54:31.892877968 (61h18m41.615441008s) [upload=58786]: Waiting for error group managing segments... Jul 27 00:54:32.223482398 (61h18m41.946045438s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.237060637 (61h18m41.959623677s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.299221539 (61h18m42.021784619s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325716521 (61h18m42.048279561s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325884473 (61h18m42.048447503s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.372983466 (61h18m42.095546506s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.379844103 (61h18m42.102407133s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.384666850 (61h18m42.10722989s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.472371113 (61h18m42.194934143s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:33.667049984 (61h18m43.389613024s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:34.213712174 (61h18m43.936275214s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:42.252927986 (61h18m51.975491026s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:59.328639826 (61h19m9.051202866s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328713146 (61h19m9.051276186s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328897458 (61h19m9.051460488s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 188.166.225.170:42398->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329496955 (61h19m9.052059995s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329528055 (61h19m9.052091095s) [seg_pos={0 0}, upload=58786]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:54:59.557689493 (61h19m9.280252533s) [upload=58786]: Done uploading segments... Jul 27 00:54:59.557710713 (61h19m9.280273753s) [upload=58786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.230742631s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 53 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......XXXXX] duration: 1m0.236188654s FAIL (2025-07-27 00:54:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== ploading piece... Jul 27 00:53:53.168746208 (61h18m2.891309248s) [noise=true , address=79.127.201.215:4354 , piece_id=FC5GGX6S7KU7UJH3L5JKNBZDBSJ73UWZPS74IWTQME5HA3Z6Z4IA, seg_pos={0 0}, upload=58775]: Uploading piece... Jul 27 00:53:53.168977931 (61h18m2.891540971s) [noise=true , address=79.127.220.101:4041 , piece_id=S7MHAFMMOTYO45KEBGB2NVAPS2RHRACNJNH6U3HXZGI5HAWHVUOA, seg_pos={0 0}, upload=58775]: Uploading piece... Jul 27 00:53:53.169161342 (61h18m2.891724433s) [noise=true , address=45.91.116.14:4206 , piece_id=XUSE6ENYXOGVMJL455NWXBKXBMMDFO74FAI2P72OBOR2W4PGNI7Q, seg_pos={0 0}, upload=58775]: Uploading piece... Jul 27 00:53:53.169285910 (61h18m2.89184894s) [noise=true , address=104.36.82.46:4513 , piece_id=QJ7YNIPOMOTVO4QPJXJEAWDIJDDM24IDEXL4RMPMJYAZH6EXZRQQ, seg_pos={0 0}, upload=58775]: Uploading piece... Jul 27 00:53:53.169573351 (61h18m2.892136381s) [noise=true , address=104.36.82.47:4538 , piece_id=4N5G2NZNXFNDB2MIEENZEWQHLRFDIWAQ3BLR7ESAYBIQC47X7HGA, seg_pos={0 0}, upload=58775]: Uploading piece... Jul 27 00:53:53.169700413 (61h18m2.892263463s) [noise=true , address=77.90.31.11:4577 , piece_id=FHFKTAKRNQEDDWZIYMPS7AAJCOTKU2HRTDUDHVDYIIHYAWSVN5UQ, seg_pos={0 0}, upload=58775]: Uploading piece... Jul 27 00:53:53.170017009 (61h18m2.89258005s) [noise=true , address=104.36.82.56:4233 , piece_id=SVOY6VF5G7IXMURYIPAJLC3NP3MD3AP4WM4NI2YQEWRMIN2MZ3FA, seg_pos={0 0}, upload=58775]: Uploading piece... Jul 27 00:53:53.170306813 (61h18m2.892869853s) [noise=true , address=104.36.82.54:4170 , piece_id=LWT7XVAHQS6NWW5TVGWN3YTKBVZVL4LWENQJCIFGRILU37Q565ZQ, seg_pos={0 0}, upload=58775]: Uploading piece... Jul 27 00:53:53.170459205 (61h18m2.893022236s) [noise=true , address=45.91.116.9:4277 , piece_id=RZUDA4VPV6AA6O6S3N5VUCNR5VL4YXNM6OCJMGQTAASB7B7FD2ZQ, seg_pos={0 0}, upload=58775]: Uploading piece... Jul 27 00:53:53.170535492 (61h18m2.893098532s) [noise=true , address=79.127.205.235:4689 , piece_id=7KEUGQUIRMO4U6ZTLSNEXCAHPT4XFZIPT35DYWI4W42JKCIUYD6Q, seg_pos={0 0}, upload=58775]: Uploading piece... Jul 27 00:53:53.170763598 (61h18m2.893326638s) [noise=true , address=79.127.205.246:4275 , piece_id=QRPYZ6SSICICG4OIL5ETHOGURLCE6GX6IOKBCY6OJBJ2PWS64COQ, seg_pos={0 0}, upload=58775]: Uploading piece... Jul 27 00:53:53.170774358 (61h18m2.893337398s) [noise=true , address=104.36.82.53:4142 , piece_id=MASVGXLOR35A7PMPJZVAUQZ7VGNW2MPS6QK6IKM5X43MR5WTIK3Q, seg_pos={0 0}, upload=58775]: Uploading piece... Jul 27 00:53:53.170990532 (61h18m2.893553562s) [noise=true , address=104.36.82.12:4156 , piece_id=5SOZ36BXIGVL4F4KW6DH7MMEGAZVE7KF4GKAVLR6XMTZ7JKYG6SA, seg_pos={0 0}, upload=58775]: Uploading piece... Jul 27 00:53:53.171193451 (61h18m2.893756491s) [noise=true , address=77.90.31.19:4875 , piece_id=T6NLO673QYCAAZXJC3EDAFKROD4XIWOHW7HKHSVGYDJGBM2VRVBQ, seg_pos={0 0}, upload=58775]: Uploading piece... Jul 27 00:53:53.172011235 (61h18m2.894574275s) [noise=true , address=156.146.43.225:4448 , piece_id=CMK5GFGZWLI5YGV2434F7E2ZCXZ26WXAGTRQLJJ5YW54XUBERGBQ, seg_pos={0 0}, upload=58775]: Uploading piece... Jul 27 00:53:53.172302743 (61h18m2.894865773s) [noise=true , address=64.49.13.138:4483 , piece_id=WWFGHEI2U7LV5QE7YK7G6WIJPNVD2HVWBEDPHXTLPGNBRTTYKGGA, seg_pos={0 0}, upload=58775]: Uploading piece... Jul 27 00:53:54.241724391 (61h18m3.964287421s) [noise=true , address=77.90.31.19:4875 , piece_id=T6NLO673QYCAAZXJC3EDAFKROD4XIWOHW7HKHSVGYDJGBM2VRVBQ, seg_pos={0 0}, upload=58775]: Done uploading piece. err: ecclient: upload failed (node:1xSFQaW3aVnAVDYE5r5ADmCGU95vWnshuZyte32W2YAj94avNn, address:77.90.31.19:4875): protocol: storage node overloaded, request limit: 2000 Jul 27 00:53:54.296211560 (61h18m4.0187746s) [upload=58775]: Next returned nil segment Jul 27 00:53:54.296233953 (61h18m4.018796993s) [upload=58775]: Waiting for error group managing segments... Jul 27 00:53:54.662346901 (61h18m4.384909941s) [noise=true , address=156.146.43.225:4448 , piece_id=CMK5GFGZWLI5YGV2434F7E2ZCXZ26WXAGTRQLJJ5YW54XUBERGBQ, seg_pos={0 0}, upload=58775]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 27 00:53:54.931601653 (61h18m4.654164693s) [noise=true , address=79.127.205.246:4275 , piece_id=QRPYZ6SSICICG4OIL5ETHOGURLCE6GX6IOKBCY6OJBJ2PWS64COQ, seg_pos={0 0}, upload=58775]: Done uploading piece. err: <nil> Jul 27 00:53:55.091837179 (61h18m4.814400209s) [noise=true , address=77.90.31.11:4577 , piece_id=FHFKTAKRNQEDDWZIYMPS7AAJCOTKU2HRTDUDHVDYIIHYAWSVN5UQ, seg_pos={0 0}, upload=58775]: Done uploading piece. err: <nil> Jul 27 00:53:55.500691554 (61h18m5.223254594s) [noise=true , address=79.127.205.235:4689 , piece_id=7KEUGQUIRMO4U6ZTLSNEXCAHPT4XFZIPT35DYWI4W42JKCIUYD6Q, seg_pos={0 0}, upload=58775]: Done uploading piece. err: <nil> Jul 27 00:53:56.604006401 (61h18m6.326569431s) [noise=true , address=79.127.220.101:4041 , piece_id=S7MHAFMMOTYO45KEBGB2NVAPS2RHRACNJNH6U3HXZGI5HAWHVUOA, seg_pos={0 0}, upload=58775]: Done uploading piece. err: <nil> Jul 27 00:53:57.983413459 (61h18m7.705976499s) [noise=true , address=64.49.13.138:4483 , piece_id=WWFGHEI2U7LV5QE7YK7G6WIJPNVD2HVWBEDPHXTLPGNBRTTYKGGA, seg_pos={0 0}, upload=58775]: Done uploading piece. err: <nil> Jul 27 00:53:59.305631086 (61h18m9.028194116s) [noise=true , address=45.91.116.14:4206 , piece_id=XUSE6ENYXOGVMJL455NWXBKXBMMDFO74FAI2P72OBOR2W4PGNI7Q, seg_pos={0 0}, upload=58775]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 27 00:54:05.729311751 (61h18m15.451874791s) [noise=true , address=45.91.116.9:4277 , piece_id=RZUDA4VPV6AA6O6S3N5VUCNR5VL4YXNM6OCJMGQTAASB7B7FD2ZQ, seg_pos={0 0}, upload=58775]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: storage node overloaded, request limit: 2000 Jul 27 00:54:07.085377531 (61h18m16.807940571s) [noise=true , address=79.127.219.38:4018 , piece_id=TPT4535I2KLRMU3GK5CN3P3XELC5FQBF2REV2EE3IGT7XCLLGAUA, seg_pos={0 0}, upload=58775]: Done uploading piece. err: <nil> Jul 27 00:54:15.234542837 (61h18m24.957105877s) [noise=true , address=79.127.201.215:4354 , piece_id=FC5GGX6S7KU7UJH3L5JKNBZDBSJ73UWZPS74IWTQME5HA3Z6Z4IA, seg_pos={0 0}, upload=58775]: Done uploading piece. err: <nil> Jul 27 00:54:30.048663660 (61h18m39.771226701s) [noise=true , address=77.90.31.18:4817 , piece_id=W3BXLGBNWGFOX725FCPXWIGTVIVP2EA64PL35UA7HXBH5ZP5ST6A, seg_pos={0 0}, upload=58775]: Done uploading piece. err: <nil> Jul 27 00:54:52.084501185 (61h19m1.807064215s) [noise=true , address=104.36.82.53:4142 , piece_id=MASVGXLOR35A7PMPJZVAUQZ7VGNW2MPS6QK6IKM5X43MR5WTIK3Q, seg_pos={0 0}, upload=58775]: Done uploading piece. err: ecclient: upload failed (node:1yLPWJs1RfZgFisHUpW3wwiFsvV98yz7adKtAKAK3unq1KZCSK, address:104.36.82.53:4142): protocol: write tcp 188.166.225.170:33040->104.36.82.53:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:52.084546602 (61h19m1.807109642s) [noise=true , address=104.36.82.46:4513 , piece_id=QJ7YNIPOMOTVO4QPJXJEAWDIJDDM24IDEXL4RMPMJYAZH6EXZRQQ, seg_pos={0 0}, upload=58775]: Done uploading piece. err: ecclient: upload failed (node:12pyyW8Gdqg6j9YGMtCUBhFnoNXryvPMUy19x815dp2HV3Xqr8h, address:104.36.82.46:4513): protocol: write tcp 188.166.225.170:33190->104.36.82.46:4513: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:52.084578563 (61h19m1.807141593s) [noise=true , address=104.36.82.47:4538 , piece_id=4N5G2NZNXFNDB2MIEENZEWQHLRFDIWAQ3BLR7ESAYBIQC47X7HGA, seg_pos={0 0}, upload=58775]: Done uploading piece. err: ecclient: upload failed (node:12P3N79R3vpvkpsJ4fEd6NcJtmMpszRW5KHgfYJvkHQzvSTf3i5, address:104.36.82.47:4538): protocol: write tcp 188.166.225.170:46520->104.36.82.47:4538: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:52.084611396 (61h19m1.807174436s) [noise=true , address=104.36.82.12:4156 , piece_id=5SOZ36BXIGVL4F4KW6DH7MMEGAZVE7KF4GKAVLR6XMTZ7JKYG6SA, seg_pos={0 0}, upload=58775]: Done uploading piece. err: ecclient: upload failed (node:12JTrtKCNHFUY91Ktaa7v7eAWe4GHssKSeofrg3diu8isHJnmy4, address:104.36.82.12:4156): protocol: write tcp 188.166.225.170:45388->104.36.82.12:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:52.084643388 (61h19m1.807206428s) [noise=true , address=104.36.82.56:4233 , piece_id=SVOY6VF5G7IXMURYIPAJLC3NP3MD3AP4WM4NI2YQEWRMIN2MZ3FA, seg_pos={0 0}, upload=58775]: Done uploading piece. err: ecclient: upload failed (node:1DmQeoQWTnFhQeJFseJoNYzFyo7vZjYKwDtQVJ6pomksWExsTb, address:104.36.82.56:4233): protocol: write tcp 188.166.225.170:33290->104.36.82.56:4233: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:52.084674908 (61h19m1.807237948s) [noise=true , address=104.36.82.54:4170 , piece_id=LWT7XVAHQS6NWW5TVGWN3YTKBVZVL4LWENQJCIFGRILU37Q565ZQ, seg_pos={0 0}, upload=58775]: Done uploading piece. err: ecclient: upload failed (node:1mcYgUVVeD2xo4pDhR7ywPLRgyA7GgejeErtLiWiPTV2AhbBRX, address:104.36.82.54:4170): protocol: write tcp 188.166.225.170:45732->104.36.82.54:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:52.084689245 (61h19m1.807252285s) [seg_pos={0 0}, upload=58775]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:54:52.318356237 (61h19m2.040919277s) [upload=58775]: Done uploading segments... Jul 27 00:54:52.318383839 (61h19m2.040946889s) [upload=58775]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.236203261s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [.............XX......X...XXX..XXX..X..X.X..............X....] duration: 23.997693033s FAIL (2025-07-27 00:54:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 23.997713623s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..XXXX.....X..........X..X..X...XX..........................] duration: 6.21538069s FAIL (2025-07-27 00:54:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.215399758s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [......................X.................................X...] duration: 1.826082737s FAIL (2025-07-27 00:54:41 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.826114959s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [..............XX......X...XXX..XXX..X..X.X..............X...] duration: 52.328091238s FAIL (2025-07-27 00:54:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 52.328109683s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......................X.................................X..] duration: 2.033396386s FAIL (2025-07-27 00:54:14 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 2.033419281s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [XX..X..XX....XX..XXXXX.X.XXXXXXXXX.XXXXXXXX..X.X.X..........] duration: 5.961707706s FAIL (2025-07-27 00:54:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== seg_pos={0 0}, upload=58618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:51940->104.36.82.8:4969: use of closed network connection; context canceled Jul 27 00:44:02.225675835 (61h8m11.948238875s) [seg_pos={0 0}, upload=58618]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:44:02.225744527 (61h8m11.948307567s) [upload=58618]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:44:02.225749156 (61h8m11.948312196s) [upload=58618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:44:02.509059970 (61h8m12.23162301s) [upload=58618]: Done uploading segments... Jul 27 00:44:02.509077874 (61h8m12.231640914s) [upload=58618]: Upload finished. err: <nil> Jul 27 00:44:39.669648204 (61h8m49.392211244s) [upload=58630]: Starting upload Jul 27 00:44:39.669713028 (61h8m49.392276068s) [upload=58630]: Done starting upload Jul 27 00:44:39.669834260 (61h8m49.39239729s) [upload=58630]: Uploading segments... Jul 27 00:44:39.669948438 (61h8m49.392511478s) [upload=58630]: Got next segment. Inline: false Jul 27 00:44:39.669993214 (61h8m49.392556254s) [upload=58630]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:44:39.669998714 (61h8m49.392561754s) [upload=58630]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:44:39.906918409 (61h8m49.629481439s) [seg_pos={0 0}, upload=58630]: Begin upload segment... Jul 27 00:44:39.906959377 (61h8m49.629522407s) [seg_pos={0 0}, upload=58630]: Done begin upload segment. Jul 27 00:44:39.906991630 (61h8m49.62955467s) [noise=true , address=45.91.116.4:4937 , piece_id=NLYBLCU3K4VLCWFKGPLNGHTONV6CCX7Y5K3MTPM3GRHLKLAGM7GA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907011507 (61h8m49.629574547s) [noise=true , address=104.36.82.12:4149 , piece_id=NXBQCY5TGXWZZDM4GYHE5HVNUFLAEU7KTA3IQF2QPDCKWF5T276Q, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907355766 (61h8m49.629918806s) [noise=true , address=45.91.116.14:4190 , piece_id=5BMBSQOU4VDTAOKYJ2EHAY55BGBWOZZUQPDFQD47HW2DF2ORLGKA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907550038 (61h8m49.630113078s) [noise=true , address=45.91.116.11:4119 , piece_id=ZTCS3JFUSDI5OS4LE3CVB3RTH6MNDF6LX5TQVC7KG7G6NZDOY75A, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907614301 (61h8m49.630177341s) [noise=true , address=45.91.116.8:4045 , piece_id=NVKHAKHTS6IUW3L7KBEONIW7U6X3YNYX5PVVDNF6GMFDN4D6QWGQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907923572 (61h8m49.630486612s) [noise=true , address=77.90.31.15:4699 , piece_id=5DNLQI7KVQA6LSMSHN7BX267GCC3W2LMYWPXUH3VKOFMWACFSJUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908056056 (61h8m49.630619096s) [noise=true , address=104.36.82.4:4803 , piece_id=MNNOC3HEF7C3ENJPGK4P6Y6YAOCROIKFILJ7C5U63K4MWDHECMXA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908189431 (61h8m49.630752471s) [noise=true , address=45.91.116.12:4227 , piece_id=HIO2GNSEEYKB7TJLECYDO4ORKH4BNJFBYXDHJDXKOZSWOATJ4HAA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908418639 (61h8m49.630981679s) [noise=true , address=45.91.116.5:4297 , piece_id=S2HC4O2W5W6SU2R6A7RXEQXDPYS74J4FHIBWNB4DTNBKPQZYTVUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908627068 (61h8m49.631190098s) [noise=true , address=104.36.82.8:4969 , piece_id=57SZMUIZBISAZAB6FAUXHI4TY6DGLZIGRGG2KBG26VBT4CR5CFUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908839696 (61h8m49.631402736s) [noise=true , address=45.91.116.8:4048 , piece_id=P35BZEGEAWHOGTRF6J7GATKTW6RBMUOT56J5DFX4YRIQVJUGJPJA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909044928 (61h8m49.631607968s) [noise=true , address=104.36.82.9:4006 , piece_id=SZSXMIL2N7P7DNGZW6MSKK6SFIVWUTQADU6RNI3AAGVZYMOJ2EKA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909265650 (61h8m49.63182869s) [noise=true , address=104.36.82.12:4150 , piece_id=QZTCPCAXF2LRJ7QIDWRK35CZEZ3VTZ2S6Y7UWPJ2UMNHW2IPHQZQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909493406 (61h8m49.632056446s) [noise=true , address=104.36.82.6:4875 , piece_id=W5L23TO2POIULWAZR77REJMF5ELCRXHIDDKDLWS6THTDLT2ZAQZQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909692708 (61h8m49.632255748s) [noise=true , address=45.91.116.10:4081 , piece_id=H7ECBWZIOD3X4MDZIREJIYXVGBVLF45RXXJQ6OJR63IMSA7MO4XQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.910010295 (61h8m49.632573335s) [noise=true , address=77.90.31.14:4661 , piece_id=MNQA3JE764QDX4DVALBLMHVBHDJNDR53KRRQO3EM26MCNCIY77EA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:41.604649775 (61h8m51.327212805s) [upload=58630]: Next returned nil segment Jul 27 00:44:41.604666106 (61h8m51.327229146s) [upload=58630]: Waiting for error group managing segments... Jul 27 00:44:41.940055171 (61h8m51.662618211s) [noise=true , address=45.91.116.5:4297 , piece_id=S2HC4O2W5W6SU2R6A7RXEQXDPYS74J4FHIBWNB4DTNBKPQZYTVUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.976252637 (61h8m51.698815677s) [noise=true , address=45.91.116.8:4045 , piece_id=NVKHAKHTS6IUW3L7KBEONIW7U6X3YNYX5PVVDNF6GMFDN4D6QWGQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.977419500 (61h8m51.69998254s) [noise=true , address=104.36.82.4:4803 , piece_id=MNNOC3HEF7C3ENJPGK4P6Y6YAOCROIKFILJ7C5U63K4MWDHECMXA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.992292375 (61h8m51.714855455s) [noise=true , address=45.91.116.12:4227 , piece_id=HIO2GNSEEYKB7TJLECYDO4ORKH4BNJFBYXDHJDXKOZSWOATJ4HAA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.005049984 (61h8m51.727613024s) [noise=true , address=45.91.116.10:4081 , piece_id=H7ECBWZIOD3X4MDZIREJIYXVGBVLF45RXXJQ6OJR63IMSA7MO4XQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.007733369 (61h8m51.730296409s) [noise=true , address=45.91.116.4:4937 , piece_id=NLYBLCU3K4VLCWFKGPLNGHTONV6CCX7Y5K3MTPM3GRHLKLAGM7GA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.015889453 (61h8m51.738452483s) [noise=true , address=77.90.31.14:4661 , piece_id=MNQA3JE764QDX4DVALBLMHVBHDJNDR53KRRQO3EM26MCNCIY77EA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.026646625 (61h8m51.749209665s) [noise=true , address=45.91.116.8:4048 , piece_id=P35BZEGEAWHOGTRF6J7GATKTW6RBMUOT56J5DFX4YRIQVJUGJPJA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.032403049 (61h8m51.754966089s) [noise=true , address=45.91.116.14:4190 , piece_id=5BMBSQOU4VDTAOKYJ2EHAY55BGBWOZZUQPDFQD47HW2DF2ORLGKA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.088809390 (61h8m51.81137243s) [noise=true , address=45.91.116.11:4119 , piece_id=ZTCS3JFUSDI5OS4LE3CVB3RTH6MNDF6LX5TQVC7KG7G6NZDOY75A, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.102686731 (61h8m51.825249771s) [noise=true , address=104.36.82.6:4875 , piece_id=W5L23TO2POIULWAZR77REJMF5ELCRXHIDDKDLWS6THTDLT2ZAQZQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.125894665 (61h8m51.848457695s) [noise=true , address=104.36.82.8:4969 , piece_id=57SZMUIZBISAZAB6FAUXHI4TY6DGLZIGRGG2KBG26VBT4CR5CFUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:45:09.023578831 (61h9m18.746141871s) [noise=true , address=104.36.82.9:4006 , piece_id=SZSXMIL2N7P7DNGZW6MSKK6SFIVWUTQADU6RNI3AAGVZYMOJ2EKA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:38976->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023652292 (61h9m18.746215332s) [noise=true , address=104.36.82.12:4150 , piece_id=QZTCPCAXF2LRJ7QIDWRK35CZEZ3VTZ2S6Y7UWPJ2UMNHW2IPHQZQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:36462->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023664465 (61h9m18.746227495s) [noise=true , address=104.36.82.12:4149 , piece_id=NXBQCY5TGXWZZDM4GYHE5HVNUFLAEU7KTA3IQF2QPDCKWF5T276Q, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:53774->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023738908 (61h9m18.746301938s) [noise=true , address=77.90.31.15:4699 , piece_id=5DNLQI7KVQA6LSMSHN7BX267GCC3W2LMYWPXUH3VKOFMWACFSJUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:47968->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023761581 (61h9m18.746324611s) [seg_pos={0 0}, upload=58630]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:45:09.265509153 (61h9m18.988072193s) [upload=58630]: Done uploading segments... Jul 27 00:45:09.265533339 (61h9m18.988096369s) [upload=58630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.961735359s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...XXXX.....X..........X..X..X...XX.........................] duration: 6.205091079s FAIL (2025-07-27 00:54:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.205111748s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 34 out of 60 is failed (threshold=3) [X...XX.XX.X.XXXXXXXXXX.X.XX..XXXXX.XXXX..XXXX.....X....X....] duration: 6.121507718s FAIL (2025-07-27 00:53:52 UTC) orbiter warning: uplink: context deadline exceeded Duration: 6.121533047s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 53 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......XXXX] duration: 1m0.244113791s FAIL (2025-07-27 00:53:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== PHCZSA, seg_pos={0 0}, upload=58758]: Uploading piece... Jul 27 00:52:52.683656822 (61h17m2.406219862s) [noise=true , address=79.127.205.234:4610 , piece_id=SLM4L5VO7YKBGXR354HH4CFX4YR2UKA457NHRIA332WMWRCFRRWQ, seg_pos={0 0}, upload=58758]: Uploading piece... Jul 27 00:52:52.683731514 (61h17m2.406294544s) [noise=true , address=77.90.31.18:4832 , piece_id=M3AZGVAKVI4NOSPRJP3WOLMUZD2PENGEPB2E7OMSFWMDGV7JMTBQ, seg_pos={0 0}, upload=58758]: Uploading piece... Jul 27 00:52:52.684050946 (61h17m2.406613986s) [noise=true , address=104.36.82.40:4969 , piece_id=D7K5A3PCX4FSWENXYLRO5AMQKH3Z5EPBAAJJPL4OLZNRTFRT6J6Q, seg_pos={0 0}, upload=58758]: Uploading piece... Jul 27 00:52:52.684307176 (61h17m2.406870216s) [noise=true , address=104.36.82.47:4530 , piece_id=BDR7KPAETWFOCFXEPCJPKYA2KEJ6HFO5XL2KXBQSUWK4JA5PKYVA, seg_pos={0 0}, upload=58758]: Uploading piece... Jul 27 00:52:52.684564869 (61h17m2.407127909s) [noise=true , address=64.49.13.26:4953 , piece_id=QF2K3Z5Y3FRTLTZ2NF757O7R65FQRAZGT63NSP3XI5OZA3YIKYIA, seg_pos={0 0}, upload=58758]: Uploading piece... Jul 27 00:52:52.684911432 (61h17m2.407474472s) [noise=true , address=79.127.205.235:4683 , piece_id=KEMSJSQY2B5H5B2VQS6F7KTRWXZ4QKS4KRNJRBCKCTIZPMIYELKA, seg_pos={0 0}, upload=58758]: Uploading piece... Jul 27 00:52:52.685163785 (61h17m2.407726825s) [noise=true , address=45.91.116.9:4277 , piece_id=FY7RVJYSIXHN22UOBY37DPLHX657MDYW66CPF2BOAA2P5U2PZKDA, seg_pos={0 0}, upload=58758]: Uploading piece... Jul 27 00:52:52.685437910 (61h17m2.40800095s) [noise=true , address=79.127.201.214:4321 , piece_id=Q6SO7ZQ2MHYJHRKKYOP7PECXSIR36UC5XNZELT3QIHIK327PQZGQ, seg_pos={0 0}, upload=58758]: Uploading piece... Jul 27 00:52:52.685581404 (61h17m2.408144444s) [noise=true , address=77.90.31.16:4766 , piece_id=7VYGV2CKMWBJMWBARHR5EISJX7PC6YITL5WG6UKWKIGDC5D5DW2Q, seg_pos={0 0}, upload=58758]: Uploading piece... Jul 27 00:52:52.685821102 (61h17m2.408384142s) [noise=true , address=104.36.82.53:4155 , piece_id=M7PMWLLOOE2LEDMSGJBJMQJBNXYXORBAAN7GD2D6ALSTEEDLQAOQ, seg_pos={0 0}, upload=58758]: Uploading piece... Jul 27 00:52:52.686029411 (61h17m2.408592451s) [noise=true , address=79.127.219.38:4009 , piece_id=HXG4CFD3DCAL75W3HZGBL7SX5ZLOCNKKQTSO46DCQYVJMMXSAACA, seg_pos={0 0}, upload=58758]: Uploading piece... Jul 27 00:52:52.686221358 (61h17m2.408784398s) [noise=true , address=104.36.82.54:4183 , piece_id=HIZOO4VSM5FDTGRHRHGLREXUA2QXDIFJWYKEYRK74KNJNWH5ORIA, seg_pos={0 0}, upload=58758]: Uploading piece... Jul 27 00:52:52.686382958 (61h17m2.408945998s) [noise=true , address=104.36.82.46:4516 , piece_id=YYAVQ53ZGHL5DICBAKQNJ7XSI2PAXN3LZRYYGJPGT3GGMF3OXHLQ, seg_pos={0 0}, upload=58758]: Uploading piece... Jul 27 00:52:52.686649548 (61h17m2.409212588s) [noise=true , address=104.36.82.55:4216 , piece_id=WCWXGZEY6ZMYZUEEWK2B7WG43Q64VQJP4WGIMN2725UDWVHFR2CQ, seg_pos={0 0}, upload=58758]: Uploading piece... Jul 27 00:52:52.686884517 (61h17m2.409447557s) [noise=true , address=45.91.116.14:4206 , piece_id=SNONYQYQC3FPX72SXNYIPXAF7UCQZPF2AQLSY44NLE4MMSHVTO6A, seg_pos={0 0}, upload=58758]: Uploading piece... Jul 27 00:52:52.687667235 (61h17m2.410230275s) [noise=true , address=77.90.31.21:4888 , piece_id=YMIKV7REP7B2KFKLZWBK5IVYWQY5DQER77HJWBE4ITGVW7ABHILA, seg_pos={0 0}, upload=58758]: Uploading piece... Jul 27 00:52:52.688060257 (61h17m2.410623287s) [noise=true , address=104.36.82.52:4108 , piece_id=2UNMFD6TDPTNUUJO5IK5CEC6Q5VVBCP25SOQYN6A7AXQN2NBPQ3Q, seg_pos={0 0}, upload=58758]: Uploading piece... Jul 27 00:52:53.797518697 (61h17m3.520081737s) [upload=58758]: Next returned nil segment Jul 27 00:52:53.797534177 (61h17m3.520097207s) [upload=58758]: Waiting for error group managing segments... Jul 27 00:52:54.097239038 (61h17m3.819802088s) [noise=true , address=79.127.205.234:4610 , piece_id=SLM4L5VO7YKBGXR354HH4CFX4YR2UKA457NHRIA332WMWRCFRRWQ, seg_pos={0 0}, upload=58758]: Done uploading piece. err: <nil> Jul 27 00:52:54.102765351 (61h17m3.825328391s) [noise=true , address=79.127.205.235:4683 , piece_id=KEMSJSQY2B5H5B2VQS6F7KTRWXZ4QKS4KRNJRBCKCTIZPMIYELKA, seg_pos={0 0}, upload=58758]: Done uploading piece. err: <nil> Jul 27 00:52:54.164818817 (61h17m3.887381857s) [noise=true , address=64.49.13.26:4953 , piece_id=QF2K3Z5Y3FRTLTZ2NF757O7R65FQRAZGT63NSP3XI5OZA3YIKYIA, seg_pos={0 0}, upload=58758]: Done uploading piece. err: <nil> Jul 27 00:52:54.209562665 (61h17m3.932125705s) [noise=true , address=79.127.201.214:4321 , piece_id=Q6SO7ZQ2MHYJHRKKYOP7PECXSIR36UC5XNZELT3QIHIK327PQZGQ, seg_pos={0 0}, upload=58758]: Done uploading piece. err: ecclient: upload failed (node:127EgxjVRhAGcz1DJjE539rWnmDjsVTRFQc1esEZUyQ1yxGiCD5, address:79.127.201.214:4321): protocol: not enough available disk space, have: 0, need: 6725632 Jul 27 00:52:54.355439710 (61h17m4.07800274s) [noise=true , address=79.127.205.245:4224 , piece_id=JVEJR33NLKOBFPUNJDHHHCRDCDYPC2A3YIRRLUH2VQORFEPHCZSA, seg_pos={0 0}, upload=58758]: Done uploading piece. err: <nil> Jul 27 00:52:54.607617929 (61h17m4.330180959s) [noise=true , address=77.90.31.18:4832 , piece_id=M3AZGVAKVI4NOSPRJP3WOLMUZD2PENGEPB2E7OMSFWMDGV7JMTBQ, seg_pos={0 0}, upload=58758]: Done uploading piece. err: <nil> Jul 27 00:52:54.722869538 (61h17m4.445432588s) [noise=true , address=77.90.31.21:4888 , piece_id=YMIKV7REP7B2KFKLZWBK5IVYWQY5DQER77HJWBE4ITGVW7ABHILA, seg_pos={0 0}, upload=58758]: Done uploading piece. err: <nil> Jul 27 00:52:54.763060895 (61h17m4.485623925s) [noise=true , address=77.90.31.16:4766 , piece_id=7VYGV2CKMWBJMWBARHR5EISJX7PC6YITL5WG6UKWKIGDC5D5DW2Q, seg_pos={0 0}, upload=58758]: Done uploading piece. err: <nil> Jul 27 00:52:56.077736424 (61h17m5.800299464s) [noise=true , address=79.127.219.38:4009 , piece_id=HXG4CFD3DCAL75W3HZGBL7SX5ZLOCNKKQTSO46DCQYVJMMXSAACA, seg_pos={0 0}, upload=58758]: Done uploading piece. err: <nil> Jul 27 00:52:57.203636802 (61h17m6.926199852s) [noise=true , address=45.91.116.9:4277 , piece_id=FY7RVJYSIXHN22UOBY37DPLHX657MDYW66CPF2BOAA2P5U2PZKDA, seg_pos={0 0}, upload=58758]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: storage node overloaded, request limit: 2000 Jul 27 00:53:01.114822875 (61h17m10.837385905s) [noise=true , address=45.91.116.14:4206 , piece_id=SNONYQYQC3FPX72SXNYIPXAF7UCQZPF2AQLSY44NLE4MMSHVTO6A, seg_pos={0 0}, upload=58758]: Done uploading piece. err: <nil> Jul 27 00:53:51.582794871 (61h18m1.305357911s) [noise=true , address=104.36.82.53:4155 , piece_id=M7PMWLLOOE2LEDMSGJBJMQJBNXYXORBAAN7GD2D6ALSTEEDLQAOQ, seg_pos={0 0}, upload=58758]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:53:51.582851198 (61h18m1.305414238s) [noise=true , address=104.36.82.47:4530 , piece_id=BDR7KPAETWFOCFXEPCJPKYA2KEJ6HFO5XL2KXBQSUWK4JA5PKYVA, seg_pos={0 0}, upload=58758]: Done uploading piece. err: ecclient: upload failed (node:1WVxMFx9jm6CT7GcXg2qgq3Be9UPFf1KZKssqufSk2ebdBBmxR, address:104.36.82.47:4530): protocol: write tcp 188.166.225.170:47834->104.36.82.47:4530: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:53:51.582941662 (61h18m1.305504702s) [noise=true , address=104.36.82.54:4183 , piece_id=HIZOO4VSM5FDTGRHRHGLREXUA2QXDIFJWYKEYRK74KNJNWH5ORIA, seg_pos={0 0}, upload=58758]: Done uploading piece. err: ecclient: upload failed (node:1kovWN9SiUhcGQtZitjH9zgPj7iHaZwY5v37bjwPvArmgX44jA, address:104.36.82.54:4183): protocol: rpc: dial tcp 104.36.82.54:4183: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:53:51.582967221 (61h18m1.305530251s) [noise=true , address=104.36.82.52:4108 , piece_id=2UNMFD6TDPTNUUJO5IK5CEC6Q5VVBCP25SOQYN6A7AXQN2NBPQ3Q, seg_pos={0 0}, upload=58758]: Done uploading piece. err: ecclient: upload failed (node:12XJJZCUSHUS52YC6jhoPB4ayjsyKZT477PymPj4inkZtdUvheT, address:104.36.82.52:4108): protocol: rpc: dial tcp 104.36.82.52:4108: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:53:51.582980676 (61h18m1.305543706s) [noise=true , address=104.36.82.55:4216 , piece_id=WCWXGZEY6ZMYZUEEWK2B7WG43Q64VQJP4WGIMN2725UDWVHFR2CQ, seg_pos={0 0}, upload=58758]: Done uploading piece. err: ecclient: upload failed (node:12vYNo8tVKDfSphJzsryTLhezUDVw3DA1NaK7ZXcXYcEw4pf1Pt, address:104.36.82.55:4216): protocol: rpc: dial tcp 104.36.82.55:4216: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:53:51.583012006 (61h18m1.305575046s) [noise=true , address=104.36.82.40:4969 , piece_id=D7K5A3PCX4FSWENXYLRO5AMQKH3Z5EPBAAJJPL4OLZNRTFRT6J6Q, seg_pos={0 0}, upload=58758]: Done uploading piece. err: ecclient: upload failed (node:12EMemJNiW3jTvHZR5KBKLoewfpdiosMtRxPkk7rtFoq66c1aR8, address:104.36.82.40:4969): protocol: write tcp 188.166.225.170:53206->104.36.82.40:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:53:51.583026364 (61h18m1.305589404s) [noise=true , address=104.36.82.46:4516 , piece_id=YYAVQ53ZGHL5DICBAKQNJ7XSI2PAXN3LZRYYGJPGT3GGMF3OXHLQ, seg_pos={0 0}, upload=58758]: Done uploading piece. err: ecclient: upload failed (node:12FRYABJbf6iHTm3WiYcL192kFth6rHjrcMAS3ahF8ddm4ox6Uj, address:104.36.82.46:4516): protocol: rpc: dial tcp 104.36.82.46:4516: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:53:51.583047585 (61h18m1.305610625s) [seg_pos={0 0}, upload=58758]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 27 00:53:51.825741247 (61h18m1.548304287s) [upload=58758]: Done uploading segments... Jul 27 00:53:51.825762407 (61h18m1.548325447s) [upload=58758]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.244125193s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 35 out of 60 is failed (threshold=3) [XX...XX.XX.X.XXXXXXXXXX.X.XX..XXXXX.XXXX..XXXX.....X....X...] duration: 5.767326568s FAIL (2025-07-27 00:53:08 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.767349513s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [........................X.................................X.] duration: 1.87946375s FAIL (2025-07-27 00:53:06 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.879486984s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X...XXXX.....X..........X..X..X...XX........................] duration: 6.149355339s FAIL (2025-07-27 00:53:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.149374606s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 34 out of 60 is failed (threshold=3) [XXX..X..XX....XX..XXXXX.X.XXXXXXXXX.XXXXXXXX..X.X.X.........] duration: 5.928355973s FAIL (2025-07-27 00:53:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== seg_pos={0 0}, upload=58618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:51940->104.36.82.8:4969: use of closed network connection; context canceled Jul 27 00:44:02.225675835 (61h8m11.948238875s) [seg_pos={0 0}, upload=58618]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:44:02.225744527 (61h8m11.948307567s) [upload=58618]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:44:02.225749156 (61h8m11.948312196s) [upload=58618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:44:02.509059970 (61h8m12.23162301s) [upload=58618]: Done uploading segments... Jul 27 00:44:02.509077874 (61h8m12.231640914s) [upload=58618]: Upload finished. err: <nil> Jul 27 00:44:39.669648204 (61h8m49.392211244s) [upload=58630]: Starting upload Jul 27 00:44:39.669713028 (61h8m49.392276068s) [upload=58630]: Done starting upload Jul 27 00:44:39.669834260 (61h8m49.39239729s) [upload=58630]: Uploading segments... Jul 27 00:44:39.669948438 (61h8m49.392511478s) [upload=58630]: Got next segment. Inline: false Jul 27 00:44:39.669993214 (61h8m49.392556254s) [upload=58630]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:44:39.669998714 (61h8m49.392561754s) [upload=58630]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:44:39.906918409 (61h8m49.629481439s) [seg_pos={0 0}, upload=58630]: Begin upload segment... Jul 27 00:44:39.906959377 (61h8m49.629522407s) [seg_pos={0 0}, upload=58630]: Done begin upload segment. Jul 27 00:44:39.906991630 (61h8m49.62955467s) [noise=true , address=45.91.116.4:4937 , piece_id=NLYBLCU3K4VLCWFKGPLNGHTONV6CCX7Y5K3MTPM3GRHLKLAGM7GA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907011507 (61h8m49.629574547s) [noise=true , address=104.36.82.12:4149 , piece_id=NXBQCY5TGXWZZDM4GYHE5HVNUFLAEU7KTA3IQF2QPDCKWF5T276Q, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907355766 (61h8m49.629918806s) [noise=true , address=45.91.116.14:4190 , piece_id=5BMBSQOU4VDTAOKYJ2EHAY55BGBWOZZUQPDFQD47HW2DF2ORLGKA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907550038 (61h8m49.630113078s) [noise=true , address=45.91.116.11:4119 , piece_id=ZTCS3JFUSDI5OS4LE3CVB3RTH6MNDF6LX5TQVC7KG7G6NZDOY75A, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907614301 (61h8m49.630177341s) [noise=true , address=45.91.116.8:4045 , piece_id=NVKHAKHTS6IUW3L7KBEONIW7U6X3YNYX5PVVDNF6GMFDN4D6QWGQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907923572 (61h8m49.630486612s) [noise=true , address=77.90.31.15:4699 , piece_id=5DNLQI7KVQA6LSMSHN7BX267GCC3W2LMYWPXUH3VKOFMWACFSJUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908056056 (61h8m49.630619096s) [noise=true , address=104.36.82.4:4803 , piece_id=MNNOC3HEF7C3ENJPGK4P6Y6YAOCROIKFILJ7C5U63K4MWDHECMXA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908189431 (61h8m49.630752471s) [noise=true , address=45.91.116.12:4227 , piece_id=HIO2GNSEEYKB7TJLECYDO4ORKH4BNJFBYXDHJDXKOZSWOATJ4HAA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908418639 (61h8m49.630981679s) [noise=true , address=45.91.116.5:4297 , piece_id=S2HC4O2W5W6SU2R6A7RXEQXDPYS74J4FHIBWNB4DTNBKPQZYTVUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908627068 (61h8m49.631190098s) [noise=true , address=104.36.82.8:4969 , piece_id=57SZMUIZBISAZAB6FAUXHI4TY6DGLZIGRGG2KBG26VBT4CR5CFUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908839696 (61h8m49.631402736s) [noise=true , address=45.91.116.8:4048 , piece_id=P35BZEGEAWHOGTRF6J7GATKTW6RBMUOT56J5DFX4YRIQVJUGJPJA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909044928 (61h8m49.631607968s) [noise=true , address=104.36.82.9:4006 , piece_id=SZSXMIL2N7P7DNGZW6MSKK6SFIVWUTQADU6RNI3AAGVZYMOJ2EKA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909265650 (61h8m49.63182869s) [noise=true , address=104.36.82.12:4150 , piece_id=QZTCPCAXF2LRJ7QIDWRK35CZEZ3VTZ2S6Y7UWPJ2UMNHW2IPHQZQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909493406 (61h8m49.632056446s) [noise=true , address=104.36.82.6:4875 , piece_id=W5L23TO2POIULWAZR77REJMF5ELCRXHIDDKDLWS6THTDLT2ZAQZQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909692708 (61h8m49.632255748s) [noise=true , address=45.91.116.10:4081 , piece_id=H7ECBWZIOD3X4MDZIREJIYXVGBVLF45RXXJQ6OJR63IMSA7MO4XQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.910010295 (61h8m49.632573335s) [noise=true , address=77.90.31.14:4661 , piece_id=MNQA3JE764QDX4DVALBLMHVBHDJNDR53KRRQO3EM26MCNCIY77EA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:41.604649775 (61h8m51.327212805s) [upload=58630]: Next returned nil segment Jul 27 00:44:41.604666106 (61h8m51.327229146s) [upload=58630]: Waiting for error group managing segments... Jul 27 00:44:41.940055171 (61h8m51.662618211s) [noise=true , address=45.91.116.5:4297 , piece_id=S2HC4O2W5W6SU2R6A7RXEQXDPYS74J4FHIBWNB4DTNBKPQZYTVUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.976252637 (61h8m51.698815677s) [noise=true , address=45.91.116.8:4045 , piece_id=NVKHAKHTS6IUW3L7KBEONIW7U6X3YNYX5PVVDNF6GMFDN4D6QWGQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.977419500 (61h8m51.69998254s) [noise=true , address=104.36.82.4:4803 , piece_id=MNNOC3HEF7C3ENJPGK4P6Y6YAOCROIKFILJ7C5U63K4MWDHECMXA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.992292375 (61h8m51.714855455s) [noise=true , address=45.91.116.12:4227 , piece_id=HIO2GNSEEYKB7TJLECYDO4ORKH4BNJFBYXDHJDXKOZSWOATJ4HAA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.005049984 (61h8m51.727613024s) [noise=true , address=45.91.116.10:4081 , piece_id=H7ECBWZIOD3X4MDZIREJIYXVGBVLF45RXXJQ6OJR63IMSA7MO4XQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.007733369 (61h8m51.730296409s) [noise=true , address=45.91.116.4:4937 , piece_id=NLYBLCU3K4VLCWFKGPLNGHTONV6CCX7Y5K3MTPM3GRHLKLAGM7GA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.015889453 (61h8m51.738452483s) [noise=true , address=77.90.31.14:4661 , piece_id=MNQA3JE764QDX4DVALBLMHVBHDJNDR53KRRQO3EM26MCNCIY77EA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.026646625 (61h8m51.749209665s) [noise=true , address=45.91.116.8:4048 , piece_id=P35BZEGEAWHOGTRF6J7GATKTW6RBMUOT56J5DFX4YRIQVJUGJPJA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.032403049 (61h8m51.754966089s) [noise=true , address=45.91.116.14:4190 , piece_id=5BMBSQOU4VDTAOKYJ2EHAY55BGBWOZZUQPDFQD47HW2DF2ORLGKA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.088809390 (61h8m51.81137243s) [noise=true , address=45.91.116.11:4119 , piece_id=ZTCS3JFUSDI5OS4LE3CVB3RTH6MNDF6LX5TQVC7KG7G6NZDOY75A, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.102686731 (61h8m51.825249771s) [noise=true , address=104.36.82.6:4875 , piece_id=W5L23TO2POIULWAZR77REJMF5ELCRXHIDDKDLWS6THTDLT2ZAQZQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.125894665 (61h8m51.848457695s) [noise=true , address=104.36.82.8:4969 , piece_id=57SZMUIZBISAZAB6FAUXHI4TY6DGLZIGRGG2KBG26VBT4CR5CFUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:45:09.023578831 (61h9m18.746141871s) [noise=true , address=104.36.82.9:4006 , piece_id=SZSXMIL2N7P7DNGZW6MSKK6SFIVWUTQADU6RNI3AAGVZYMOJ2EKA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:38976->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023652292 (61h9m18.746215332s) [noise=true , address=104.36.82.12:4150 , piece_id=QZTCPCAXF2LRJ7QIDWRK35CZEZ3VTZ2S6Y7UWPJ2UMNHW2IPHQZQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:36462->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023664465 (61h9m18.746227495s) [noise=true , address=104.36.82.12:4149 , piece_id=NXBQCY5TGXWZZDM4GYHE5HVNUFLAEU7KTA3IQF2QPDCKWF5T276Q, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:53774->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023738908 (61h9m18.746301938s) [noise=true , address=77.90.31.15:4699 , piece_id=5DNLQI7KVQA6LSMSHN7BX267GCC3W2LMYWPXUH3VKOFMWACFSJUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:47968->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023761581 (61h9m18.746324611s) [seg_pos={0 0}, upload=58630]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:45:09.265509153 (61h9m18.988072193s) [upload=58630]: Done uploading segments... Jul 27 00:45:09.265533339 (61h9m18.988096369s) [upload=58630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.928373447s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [...............XX......X...XXX..XXX..X..X.X..............X..] duration: 46.816448374s FAIL (2025-07-27 00:53:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 46.816473101s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 53 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......XXX] duration: 1m0.237669134s FAIL (2025-07-27 00:52:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== .193248496 (61h16m1.915811526s) [noise=true , address=104.36.82.47:4528 , piece_id=JPSCUGUXHSXSPBETK2GCH3HSUUAXG6GSKFPQHIXQC2SSZWFDVF7Q, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:51:52.193540344 (61h16m1.916103384s) [noise=true , address=79.127.219.38:4018 , piece_id=RLD6X64O5H7PZDHXJHGBT4NNUHJVDK3T2ARCLXOJKU56WUDKYEFA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:51:52.193741530 (61h16m1.91630457s) [noise=true , address=156.146.43.225:4449 , piece_id=Q7C735TQAP257QC4S34I2TYKGYJ6U37TL7X2MOLIVGJ6T2SJC3EQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:51:52.193873852 (61h16m1.916436892s) [noise=true , address=104.36.82.46:4484 , piece_id=WEQO7ZM7NAIZH3FFZCHJMXA6SB5L3G3N6K4QAVG324QN2PWYQL5Q, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:51:52.194032927 (61h16m1.916595967s) [noise=true , address=79.127.205.234:4605 , piece_id=6NI3B5NYWROZTUPQFKGNC55B4QB3O7ATGVN7ZEVT7JJMZQJUKKBQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:51:52.194164378 (61h16m1.916727408s) [noise=true , address=45.91.116.10:4103 , piece_id=UGYYMJHOB4KCFIVKP67RN63RLMFOBAEIURNBSX5OXVPHGVJA6PFQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:51:52.194317751 (61h16m1.916880801s) [noise=true , address=79.127.201.215:4347 , piece_id=KTKDIRKTLA6MU2XHZBR2UT34LUHUO3BGQKXILS2UXOEHAA2RHSYA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:51:52.194619649 (61h16m1.917182699s) [noise=true , address=104.36.82.50:4019 , piece_id=JQW5M27ANQ74XXB6HRFRWJLK6UCGSMDL4IRVFLSSTG5SBFUNWIOQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:51:52.194793161 (61h16m1.917356202s) [noise=true , address=104.36.82.53:4141 , piece_id=F7QU2DVANWTVXTH724DIOJOPXJ2G5FQEGGCS5MJWRZEPDJ45EAFQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:51:52.194884736 (61h16m1.917447776s) [noise=true , address=104.36.82.52:4119 , piece_id=2SDDXSBLU663OX5ISSLDIPIW3KSNJ3MZEBPYEN3RK7NC7YXDQUVQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:51:52.195147439 (61h16m1.91771048s) [noise=true , address=79.127.205.245:4236 , piece_id=5ZHBDLI7FJWYR67L7MKQCI7CDBX437KYG6VBLSUNMAZ65D5YWOMA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:51:52.195154182 (61h16m1.917717222s) [noise=true , address=79.127.205.246:4282 , piece_id=VTN32X6VORAVZXRNHU2R2KIMHGVZ2ZN4YBQU7V4SE4RLE7PEWGSQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:51:52.195496256 (61h16m1.918059316s) [noise=true , address=77.90.31.21:4912 , piece_id=ZNK3WZQ7APPOV5UUMARJO6CSKQNEFERXPX5XTTW3MGVL3TFHAGGA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:51:52.195572863 (61h16m1.918135913s) [noise=true , address=104.36.82.12:4158 , piece_id=MSS5NQAAXHEWVX4KE2WRKZ7A2DTPHWQ255NK6CO3FMLDBODMUZCQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:51:52.195911901 (61h16m1.918474941s) [noise=true , address=104.36.82.41:4768 , piece_id=GSMNADDDOFVI4FQY6OBYXP3MWJXGDDBUF2MVAKGUUMT2HO6WIO2Q, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:51:52.196006142 (61h16m1.918569182s) [noise=true , address=104.36.82.55:4207 , piece_id=UUZYLF5GU5FGPUVP4Y5FMXMZH6DVXYAKN4C4GWDNMQBT3JOR2JMA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:51:53.200374847 (61h16m2.922937887s) [noise=true , address=156.146.43.225:4449 , piece_id=Q7C735TQAP257QC4S34I2TYKGYJ6U37TL7X2MOLIVGJ6T2SJC3EQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:12sjnjDAH1FS9CNxSfYy3gBL2e19uQoyeWVeGtSFVcpRK9SYYZk, address:156.146.43.225:4449): protocol: storage node overloaded, request limit: 2000 Jul 27 00:51:53.311220175 (61h16m3.033783205s) [upload=58744]: Next returned nil segment Jul 27 00:51:53.311237679 (61h16m3.033800709s) [upload=58744]: Waiting for error group managing segments... Jul 27 00:51:53.618653624 (61h16m3.341216654s) [noise=true , address=79.127.205.234:4605 , piece_id=6NI3B5NYWROZTUPQFKGNC55B4QB3O7ATGVN7ZEVT7JJMZQJUKKBQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:51:53.619367910 (61h16m3.341930951s) [noise=true , address=79.127.205.246:4282 , piece_id=VTN32X6VORAVZXRNHU2R2KIMHGVZ2ZN4YBQU7V4SE4RLE7PEWGSQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:51:53.635876536 (61h16m3.358439576s) [noise=true , address=79.127.205.245:4236 , piece_id=5ZHBDLI7FJWYR67L7MKQCI7CDBX437KYG6VBLSUNMAZ65D5YWOMA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:51:54.134787915 (61h16m3.857350986s) [noise=true , address=77.90.31.18:4812 , piece_id=NXMBZGKEKGLO4PJ42ZEYSNDOJQX64NLKOAXMI54LV4U25SJGTIDA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:51:54.271926200 (61h16m3.99448924s) [noise=true , address=45.91.116.10:4103 , piece_id=UGYYMJHOB4KCFIVKP67RN63RLMFOBAEIURNBSX5OXVPHGVJA6PFQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:51:54.300910415 (61h16m4.023473455s) [noise=true , address=77.90.31.21:4912 , piece_id=ZNK3WZQ7APPOV5UUMARJO6CSKQNEFERXPX5XTTW3MGVL3TFHAGGA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:05.327439808 (61h16m15.050002848s) [noise=true , address=79.127.219.38:4018 , piece_id=RLD6X64O5H7PZDHXJHGBT4NNUHJVDK3T2ARCLXOJKU56WUDKYEFA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:21.677598147 (61h16m31.400161187s) [noise=true , address=79.127.201.215:4347 , piece_id=KTKDIRKTLA6MU2XHZBR2UT34LUHUO3BGQKXILS2UXOEHAA2RHSYA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:51.114225222 (61h17m0.836788262s) [noise=true , address=104.36.82.53:4141 , piece_id=F7QU2DVANWTVXTH724DIOJOPXJ2G5FQEGGCS5MJWRZEPDJ45EAFQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:15wUF135h4QsyvF7Cr3eb6rxtx8KPz1LHbgRjQYwiF5xpmE2nn, address:104.36.82.53:4141): protocol: rpc: dial tcp 104.36.82.53:4141: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:52:51.114270208 (61h17m0.836833248s) [noise=true , address=104.36.82.47:4528 , piece_id=JPSCUGUXHSXSPBETK2GCH3HSUUAXG6GSKFPQHIXQC2SSZWFDVF7Q, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:12ArcCrcHdpNgBMfTcPdtTXb65RgnTh8N32ihHAx2RrddxnRYgT, address:104.36.82.47:4528): protocol: rpc: dial tcp 104.36.82.47:4528: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:52:51.114303211 (61h17m0.836866241s) [noise=true , address=104.36.82.50:4019 , piece_id=JQW5M27ANQ74XXB6HRFRWJLK6UCGSMDL4IRVFLSSTG5SBFUNWIOQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:1ztmvXKmY6hicHZD7iejpcJCKFEhLwemtaYChAp8Po85TRQvGo, address:104.36.82.50:4019): protocol: write tcp 188.166.225.170:41968->104.36.82.50:4019: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:52:51.114332517 (61h17m0.836895557s) [noise=true , address=104.36.82.52:4119 , piece_id=2SDDXSBLU663OX5ISSLDIPIW3KSNJ3MZEBPYEN3RK7NC7YXDQUVQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:12mTLjL6Y6VVjqztwVXYif4v7tyftjLafztLUibajksMHCFraFG, address:104.36.82.52:4119): protocol: rpc: dial tcp 104.36.82.52:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:52:51.114347997 (61h17m0.836911027s) [noise=true , address=104.36.82.55:4207 , piece_id=UUZYLF5GU5FGPUVP4Y5FMXMZH6DVXYAKN4C4GWDNMQBT3JOR2JMA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: rpc: dial tcp 104.36.82.55:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:52:51.114361432 (61h17m0.836924472s) [noise=true , address=104.36.82.12:4158 , piece_id=MSS5NQAAXHEWVX4KE2WRKZ7A2DTPHWQ255NK6CO3FMLDBODMUZCQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:4158): protocol: rpc: dial tcp 104.36.82.12:4158: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:52:51.114389806 (61h17m0.836952847s) [noise=true , address=104.36.82.41:4768 , piece_id=GSMNADDDOFVI4FQY6OBYXP3MWJXGDDBUF2MVAKGUUMT2HO6WIO2Q, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4, address:104.36.82.41:4768): protocol: rpc: dial tcp 104.36.82.41:4768: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:52:51.114398213 (61h17m0.836961253s) [noise=true , address=104.36.82.15:4277 , piece_id=EMWNCHRV3R34HCQN6YHPOELNS7QRVRA46BCH6ZPXRDTYCMAHS5VQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:1i493KQZuQYQwXzct9HamWDCq92CSGvLPHjZ6fxxVVZX1f7ofY, address:104.36.82.15:4277): protocol: rpc: dial tcp 104.36.82.15:4277: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:52:51.114424624 (61h17m0.836987654s) [noise=true , address=104.36.82.46:4484 , piece_id=WEQO7ZM7NAIZH3FFZCHJMXA6SB5L3G3N6K4QAVG324QN2PWYQL5Q, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:1a5S5Ar7jNbK9MESghbWfmEgBPNSMN2tEnZeeG1kKzeX2dnSX3, address:104.36.82.46:4484): protocol: write tcp 188.166.225.170:33594->104.36.82.46:4484: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:52:51.114444581 (61h17m0.837007621s) [seg_pos={0 0}, upload=58744]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:52:51.349561367 (61h17m1.072124407s) [upload=58744]: Done uploading segments... Jul 27 00:52:51.349591346 (61h17m1.072154386s) [upload=58744]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.237690734s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [XX...XXXX.....X..........X..X..X...XX.......................] duration: 6.118895513s FAIL (2025-07-27 00:52:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.118916403s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 36 out of 60 is failed (threshold=3) [XXX...XX.XX.X.XXXXXXXXXX.X.XX..XXXXX.XXXX..XXXX.....X....X..] duration: 5.743845272s FAIL (2025-07-27 00:52:33 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.743873015s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [XXXX..X..XX....XX..XXXXX.X.XXXXXXXXX.XXXXXXXX..X.X.X........] duration: 7.684120002s FAIL (2025-07-27 00:52:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== seg_pos={0 0}, upload=58618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:51940->104.36.82.8:4969: use of closed network connection; context canceled Jul 27 00:44:02.225675835 (61h8m11.948238875s) [seg_pos={0 0}, upload=58618]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:44:02.225744527 (61h8m11.948307567s) [upload=58618]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:44:02.225749156 (61h8m11.948312196s) [upload=58618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:44:02.509059970 (61h8m12.23162301s) [upload=58618]: Done uploading segments... Jul 27 00:44:02.509077874 (61h8m12.231640914s) [upload=58618]: Upload finished. err: <nil> Jul 27 00:44:39.669648204 (61h8m49.392211244s) [upload=58630]: Starting upload Jul 27 00:44:39.669713028 (61h8m49.392276068s) [upload=58630]: Done starting upload Jul 27 00:44:39.669834260 (61h8m49.39239729s) [upload=58630]: Uploading segments... Jul 27 00:44:39.669948438 (61h8m49.392511478s) [upload=58630]: Got next segment. Inline: false Jul 27 00:44:39.669993214 (61h8m49.392556254s) [upload=58630]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:44:39.669998714 (61h8m49.392561754s) [upload=58630]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:44:39.906918409 (61h8m49.629481439s) [seg_pos={0 0}, upload=58630]: Begin upload segment... Jul 27 00:44:39.906959377 (61h8m49.629522407s) [seg_pos={0 0}, upload=58630]: Done begin upload segment. Jul 27 00:44:39.906991630 (61h8m49.62955467s) [noise=true , address=45.91.116.4:4937 , piece_id=NLYBLCU3K4VLCWFKGPLNGHTONV6CCX7Y5K3MTPM3GRHLKLAGM7GA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907011507 (61h8m49.629574547s) [noise=true , address=104.36.82.12:4149 , piece_id=NXBQCY5TGXWZZDM4GYHE5HVNUFLAEU7KTA3IQF2QPDCKWF5T276Q, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907355766 (61h8m49.629918806s) [noise=true , address=45.91.116.14:4190 , piece_id=5BMBSQOU4VDTAOKYJ2EHAY55BGBWOZZUQPDFQD47HW2DF2ORLGKA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907550038 (61h8m49.630113078s) [noise=true , address=45.91.116.11:4119 , piece_id=ZTCS3JFUSDI5OS4LE3CVB3RTH6MNDF6LX5TQVC7KG7G6NZDOY75A, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907614301 (61h8m49.630177341s) [noise=true , address=45.91.116.8:4045 , piece_id=NVKHAKHTS6IUW3L7KBEONIW7U6X3YNYX5PVVDNF6GMFDN4D6QWGQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907923572 (61h8m49.630486612s) [noise=true , address=77.90.31.15:4699 , piece_id=5DNLQI7KVQA6LSMSHN7BX267GCC3W2LMYWPXUH3VKOFMWACFSJUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908056056 (61h8m49.630619096s) [noise=true , address=104.36.82.4:4803 , piece_id=MNNOC3HEF7C3ENJPGK4P6Y6YAOCROIKFILJ7C5U63K4MWDHECMXA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908189431 (61h8m49.630752471s) [noise=true , address=45.91.116.12:4227 , piece_id=HIO2GNSEEYKB7TJLECYDO4ORKH4BNJFBYXDHJDXKOZSWOATJ4HAA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908418639 (61h8m49.630981679s) [noise=true , address=45.91.116.5:4297 , piece_id=S2HC4O2W5W6SU2R6A7RXEQXDPYS74J4FHIBWNB4DTNBKPQZYTVUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908627068 (61h8m49.631190098s) [noise=true , address=104.36.82.8:4969 , piece_id=57SZMUIZBISAZAB6FAUXHI4TY6DGLZIGRGG2KBG26VBT4CR5CFUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908839696 (61h8m49.631402736s) [noise=true , address=45.91.116.8:4048 , piece_id=P35BZEGEAWHOGTRF6J7GATKTW6RBMUOT56J5DFX4YRIQVJUGJPJA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909044928 (61h8m49.631607968s) [noise=true , address=104.36.82.9:4006 , piece_id=SZSXMIL2N7P7DNGZW6MSKK6SFIVWUTQADU6RNI3AAGVZYMOJ2EKA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909265650 (61h8m49.63182869s) [noise=true , address=104.36.82.12:4150 , piece_id=QZTCPCAXF2LRJ7QIDWRK35CZEZ3VTZ2S6Y7UWPJ2UMNHW2IPHQZQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909493406 (61h8m49.632056446s) [noise=true , address=104.36.82.6:4875 , piece_id=W5L23TO2POIULWAZR77REJMF5ELCRXHIDDKDLWS6THTDLT2ZAQZQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909692708 (61h8m49.632255748s) [noise=true , address=45.91.116.10:4081 , piece_id=H7ECBWZIOD3X4MDZIREJIYXVGBVLF45RXXJQ6OJR63IMSA7MO4XQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.910010295 (61h8m49.632573335s) [noise=true , address=77.90.31.14:4661 , piece_id=MNQA3JE764QDX4DVALBLMHVBHDJNDR53KRRQO3EM26MCNCIY77EA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:41.604649775 (61h8m51.327212805s) [upload=58630]: Next returned nil segment Jul 27 00:44:41.604666106 (61h8m51.327229146s) [upload=58630]: Waiting for error group managing segments... Jul 27 00:44:41.940055171 (61h8m51.662618211s) [noise=true , address=45.91.116.5:4297 , piece_id=S2HC4O2W5W6SU2R6A7RXEQXDPYS74J4FHIBWNB4DTNBKPQZYTVUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.976252637 (61h8m51.698815677s) [noise=true , address=45.91.116.8:4045 , piece_id=NVKHAKHTS6IUW3L7KBEONIW7U6X3YNYX5PVVDNF6GMFDN4D6QWGQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.977419500 (61h8m51.69998254s) [noise=true , address=104.36.82.4:4803 , piece_id=MNNOC3HEF7C3ENJPGK4P6Y6YAOCROIKFILJ7C5U63K4MWDHECMXA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.992292375 (61h8m51.714855455s) [noise=true , address=45.91.116.12:4227 , piece_id=HIO2GNSEEYKB7TJLECYDO4ORKH4BNJFBYXDHJDXKOZSWOATJ4HAA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.005049984 (61h8m51.727613024s) [noise=true , address=45.91.116.10:4081 , piece_id=H7ECBWZIOD3X4MDZIREJIYXVGBVLF45RXXJQ6OJR63IMSA7MO4XQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.007733369 (61h8m51.730296409s) [noise=true , address=45.91.116.4:4937 , piece_id=NLYBLCU3K4VLCWFKGPLNGHTONV6CCX7Y5K3MTPM3GRHLKLAGM7GA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.015889453 (61h8m51.738452483s) [noise=true , address=77.90.31.14:4661 , piece_id=MNQA3JE764QDX4DVALBLMHVBHDJNDR53KRRQO3EM26MCNCIY77EA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.026646625 (61h8m51.749209665s) [noise=true , address=45.91.116.8:4048 , piece_id=P35BZEGEAWHOGTRF6J7GATKTW6RBMUOT56J5DFX4YRIQVJUGJPJA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.032403049 (61h8m51.754966089s) [noise=true , address=45.91.116.14:4190 , piece_id=5BMBSQOU4VDTAOKYJ2EHAY55BGBWOZZUQPDFQD47HW2DF2ORLGKA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.088809390 (61h8m51.81137243s) [noise=true , address=45.91.116.11:4119 , piece_id=ZTCS3JFUSDI5OS4LE3CVB3RTH6MNDF6LX5TQVC7KG7G6NZDOY75A, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.102686731 (61h8m51.825249771s) [noise=true , address=104.36.82.6:4875 , piece_id=W5L23TO2POIULWAZR77REJMF5ELCRXHIDDKDLWS6THTDLT2ZAQZQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.125894665 (61h8m51.848457695s) [noise=true , address=104.36.82.8:4969 , piece_id=57SZMUIZBISAZAB6FAUXHI4TY6DGLZIGRGG2KBG26VBT4CR5CFUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:45:09.023578831 (61h9m18.746141871s) [noise=true , address=104.36.82.9:4006 , piece_id=SZSXMIL2N7P7DNGZW6MSKK6SFIVWUTQADU6RNI3AAGVZYMOJ2EKA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:38976->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023652292 (61h9m18.746215332s) [noise=true , address=104.36.82.12:4150 , piece_id=QZTCPCAXF2LRJ7QIDWRK35CZEZ3VTZ2S6Y7UWPJ2UMNHW2IPHQZQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:36462->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023664465 (61h9m18.746227495s) [noise=true , address=104.36.82.12:4149 , piece_id=NXBQCY5TGXWZZDM4GYHE5HVNUFLAEU7KTA3IQF2QPDCKWF5T276Q, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:53774->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023738908 (61h9m18.746301938s) [noise=true , address=77.90.31.15:4699 , piece_id=5DNLQI7KVQA6LSMSHN7BX267GCC3W2LMYWPXUH3VKOFMWACFSJUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:47968->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023761581 (61h9m18.746324611s) [seg_pos={0 0}, upload=58630]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:45:09.265509153 (61h9m18.988072193s) [upload=58630]: Done uploading segments... Jul 27 00:45:09.265533339 (61h9m18.988096369s) [upload=58630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.684186088s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.XX...XXXX.....X..........X..X..X...XX......................] duration: 5.822312866s FAIL (2025-07-27 00:52:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.822336511s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........................X.................................X] duration: 1.667649949s FAIL (2025-07-27 00:51:59 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.667674475s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 53 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......XX] duration: 1m0.235228042s FAIL (2025-07-27 00:51:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 6085056s) [seg_pos={0 0}, upload=58731]: Done begin upload segment. Jul 27 00:50:51.523545381 (61h15m1.246108421s) [noise=true , address=79.127.219.47:4048 , piece_id=NECWDN2WQTXNCSFLS22WMR6KYM4YG4TSMUFU4ED2M36U6EUFI32A, seg_pos={0 0}, upload=58731]: Uploading piece... Jul 27 00:50:51.523613611 (61h15m1.246176651s) [noise=true , address=79.127.201.215:4358 , piece_id=6WQULSSBB66OR2YBNUSIMZSSOMTWQ653X4ZPV4ZMJA4ZCN2DXYAQ, seg_pos={0 0}, upload=58731]: Uploading piece... Jul 27 00:50:51.524089021 (61h15m1.246652061s) [noise=true , address=77.90.31.18:4832 , piece_id=I7MBWEFGAEQF3IJUG7OOXIDXR4MAMS4SD3LNQHROXYCF4WJI3QIA, seg_pos={0 0}, upload=58731]: Uploading piece... Jul 27 00:50:51.524215432 (61h15m1.246778472s) [noise=true , address=104.36.82.47:4549 , piece_id=TMUE25DNNOMV2LKHRNDUM5BLIVQIPIYVN7N7QCBMUG4XQ3RUYZ3Q, seg_pos={0 0}, upload=58731]: Uploading piece... Jul 27 00:50:51.524376270 (61h15m1.24693931s) [noise=true , address=104.36.82.56:4233 , piece_id=BHIQILP3H4M337UWEIACAEEIMCFYZCRK7RGMSQ6WF3JXOJJR6X7Q, seg_pos={0 0}, upload=58731]: Uploading piece... Jul 27 00:50:51.524522870 (61h15m1.24708591s) [noise=true , address=77.90.31.10:4537 , piece_id=BR6JCZAAVJ6DY742NARWBBHTJMMHFLUABLRNHVHU6KER6TOFYUBQ, seg_pos={0 0}, upload=58731]: Uploading piece... Jul 27 00:50:51.524653581 (61h15m1.247216611s) [noise=true , address=104.36.82.55:4221 , piece_id=AHDB66BTWYNFWXCRRF6OGWX3ZUWF5J6FFSWTQVB5OKFCB3BZRVXQ, seg_pos={0 0}, upload=58731]: Uploading piece... Jul 27 00:50:51.524819728 (61h15m1.247382769s) [noise=true , address=45.91.116.14:4194 , piece_id=J7MML5GNDBCNSBIJKNISNHWY2SUKEFND2VVHGLKTAY5N322E3L2A, seg_pos={0 0}, upload=58731]: Uploading piece... Jul 27 00:50:51.524958975 (61h15m1.247522015s) [noise=true , address=79.127.205.245:4221 , piece_id=73TESNL5YRJBHWHZEKIMMBIVVB7YZGG7HUQNQZVWRTP6L5XHD4AQ, seg_pos={0 0}, upload=58731]: Uploading piece... Jul 27 00:50:51.525119933 (61h15m1.247682973s) [noise=true , address=104.36.82.54:4181 , piece_id=MNB2FUSAHRINEKJ4USWRTXIQH3ZEIUYCFFVMOF7OLAELOPTC5L5A, seg_pos={0 0}, upload=58731]: Uploading piece... Jul 27 00:50:51.525213291 (61h15m1.247776331s) [noise=true , address=77.90.31.16:4748 , piece_id=XX5GPXLHBB3P7GKW3VJV6TZV6AW7CD32ILKYC4AGD3RYFY2CHCKQ, seg_pos={0 0}, upload=58731]: Uploading piece... Jul 27 00:50:51.525448281 (61h15m1.248011321s) [noise=true , address=104.36.82.9:4024 , piece_id=QBP2OLQJAL2XHIYQOLDI3AWNZ5K5O6R7FZOD4O2UZY7VI3HEU7HA, seg_pos={0 0}, upload=58731]: Uploading piece... Jul 27 00:50:51.525566006 (61h15m1.248129086s) [noise=true , address=104.36.82.46:4493 , piece_id=CZL766BZTN2RMN4ANWTSAV2H7RGE7SLXT5FUOA5YBI2YSBOVJPOA, seg_pos={0 0}, upload=58731]: Uploading piece... Jul 27 00:50:51.525663373 (61h15m1.248226413s) [noise=true , address=79.127.219.38:4018 , piece_id=7V6K6TTYFHZ2PLYPGEQHFQY6PWH5MBD3OA4CKJDTD26P5XB6WNIA, seg_pos={0 0}, upload=58731]: Uploading piece... Jul 27 00:50:51.526132861 (61h15m1.248695921s) [noise=true , address=104.36.82.52:4089 , piece_id=DPLY6FMK44ZW3VF5G25QQP6ELK4XBPUG4NLW4MGXEEYR57VL3UMA, seg_pos={0 0}, upload=58731]: Uploading piece... Jul 27 00:50:51.526627798 (61h15m1.249190858s) [noise=true , address=79.127.205.235:4689 , piece_id=B4PAG2NZ3ANPMRHUUA5KI47JZEOFDBSSBV6UVCXNGQORBEBUJFKQ, seg_pos={0 0}, upload=58731]: Uploading piece... Jul 27 00:50:51.527103468 (61h15m1.249666508s) [noise=true , address=79.127.220.97:4361 , piece_id=IT7GRBERG24JD6GYSGELWV7JON7225THKRGMO6DA5CAZGL4DMCHA, seg_pos={0 0}, upload=58731]: Uploading piece... Jul 27 00:50:51.527432548 (61h15m1.249995588s) [noise=true , address=79.127.205.233:4643 , piece_id=H3NGKKUGSUUO5VVV4JA2HIWBD364PMJJQICFWNWK2QDSRVW4ZXPA, seg_pos={0 0}, upload=58731]: Uploading piece... Jul 27 00:50:52.621887713 (61h15m2.344450743s) [noise=true , address=79.127.220.97:4361 , piece_id=IT7GRBERG24JD6GYSGELWV7JON7225THKRGMO6DA5CAZGL4DMCHA, seg_pos={0 0}, upload=58731]: Done uploading piece. err: ecclient: upload failed (node:1ojv5zf5QshKyMqGypaEaYmGvzt96UM9kZgyGeFVPMFk8Twzyn, address:79.127.220.97:4361): protocol: storage node overloaded, request limit: 2000 Jul 27 00:50:52.636230003 (61h15m2.358793043s) [upload=58731]: Next returned nil segment Jul 27 00:50:52.636245824 (61h15m2.358808854s) [upload=58731]: Waiting for error group managing segments... Jul 27 00:50:52.973183140 (61h15m2.69574618s) [noise=true , address=79.127.205.235:4689 , piece_id=B4PAG2NZ3ANPMRHUUA5KI47JZEOFDBSSBV6UVCXNGQORBEBUJFKQ, seg_pos={0 0}, upload=58731]: Done uploading piece. err: <nil> Jul 27 00:50:53.029532954 (61h15m2.752095994s) [noise=true , address=79.127.205.245:4221 , piece_id=73TESNL5YRJBHWHZEKIMMBIVVB7YZGG7HUQNQZVWRTP6L5XHD4AQ, seg_pos={0 0}, upload=58731]: Done uploading piece. err: <nil> Jul 27 00:50:53.523372920 (61h15m3.24593596s) [noise=true , address=77.90.31.10:4537 , piece_id=BR6JCZAAVJ6DY742NARWBBHTJMMHFLUABLRNHVHU6KER6TOFYUBQ, seg_pos={0 0}, upload=58731]: Done uploading piece. err: <nil> Jul 27 00:50:53.529316952 (61h15m3.251879992s) [noise=true , address=77.90.31.18:4832 , piece_id=I7MBWEFGAEQF3IJUG7OOXIDXR4MAMS4SD3LNQHROXYCF4WJI3QIA, seg_pos={0 0}, upload=58731]: Done uploading piece. err: <nil> Jul 27 00:50:53.651855504 (61h15m3.374418544s) [noise=true , address=79.127.205.233:4643 , piece_id=H3NGKKUGSUUO5VVV4JA2HIWBD364PMJJQICFWNWK2QDSRVW4ZXPA, seg_pos={0 0}, upload=58731]: Done uploading piece. err: <nil> Jul 27 00:50:53.771517020 (61h15m3.49408006s) [noise=true , address=45.91.116.14:4194 , piece_id=J7MML5GNDBCNSBIJKNISNHWY2SUKEFND2VVHGLKTAY5N322E3L2A, seg_pos={0 0}, upload=58731]: Done uploading piece. err: <nil> Jul 27 00:50:54.078279374 (61h15m3.800842404s) [noise=true , address=79.127.219.47:4048 , piece_id=NECWDN2WQTXNCSFLS22WMR6KYM4YG4TSMUFU4ED2M36U6EUFI32A, seg_pos={0 0}, upload=58731]: Done uploading piece. err: <nil> Jul 27 00:50:59.562878576 (61h15m9.285441616s) [noise=true , address=79.127.219.38:4018 , piece_id=7V6K6TTYFHZ2PLYPGEQHFQY6PWH5MBD3OA4CKJDTD26P5XB6WNIA, seg_pos={0 0}, upload=58731]: Done uploading piece. err: <nil> Jul 27 00:51:11.394240632 (61h15m21.116803672s) [noise=true , address=79.127.201.215:4358 , piece_id=6WQULSSBB66OR2YBNUSIMZSSOMTWQ653X4ZPV4ZMJA4ZCN2DXYAQ, seg_pos={0 0}, upload=58731]: Done uploading piece. err: <nil> Jul 27 00:51:18.659309529 (61h15m28.381872559s) [noise=true , address=77.90.31.16:4748 , piece_id=XX5GPXLHBB3P7GKW3VJV6TZV6AW7CD32ILKYC4AGD3RYFY2CHCKQ, seg_pos={0 0}, upload=58731]: Done uploading piece. err: <nil> Jul 27 00:51:50.428318192 (61h16m0.150881232s) [noise=true , address=104.36.82.55:4221 , piece_id=AHDB66BTWYNFWXCRRF6OGWX3ZUWF5J6FFSWTQVB5OKFCB3BZRVXQ, seg_pos={0 0}, upload=58731]: Done uploading piece. err: ecclient: upload failed (node:1ssFiXk5oKTgnVMJ5TBJuSXuD1D2C46gT2hcY6yaDxf8fL1cjm, address:104.36.82.55:4221): protocol: rpc: dial tcp 104.36.82.55:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:51:50.428357608 (61h16m0.150920648s) [noise=true , address=104.36.82.46:4493 , piece_id=CZL766BZTN2RMN4ANWTSAV2H7RGE7SLXT5FUOA5YBI2YSBOVJPOA, seg_pos={0 0}, upload=58731]: Done uploading piece. err: ecclient: upload failed (node:127drPYZWm7RRXuCLtuzDgSaxxuu3k1tg6eo3KWtrQGjBaUbfRJ, address:104.36.82.46:4493): protocol: rpc: dial tcp 104.36.82.46:4493: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:51:50.428389840 (61h16m0.15095288s) [noise=true , address=104.36.82.47:4549 , piece_id=TMUE25DNNOMV2LKHRNDUM5BLIVQIPIYVN7N7QCBMUG4XQ3RUYZ3Q, seg_pos={0 0}, upload=58731]: Done uploading piece. err: ecclient: upload failed (node:1HWoyMKQ6v2XWUfEmfFh3m8AiGivAcYkkZ9na8kqHFzH2hPDeK, address:104.36.82.47:4549): protocol: rpc: dial tcp 104.36.82.47:4549: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:51:50.428409397 (61h16m0.150972437s) [noise=true , address=104.36.82.54:4181 , piece_id=MNB2FUSAHRINEKJ4USWRTXIQH3ZEIUYCFFVMOF7OLAELOPTC5L5A, seg_pos={0 0}, upload=58731]: Done uploading piece. err: ecclient: upload failed (node:12gGf5E7tzF2TXtLWgpK55t24SC1BpTSeC9G66nPmR1ARnDrAE8, address:104.36.82.54:4181): protocol: rpc: dial tcp 104.36.82.54:4181: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:51:50.428431068 (61h16m0.150994108s) [noise=true , address=104.36.82.52:4089 , piece_id=DPLY6FMK44ZW3VF5G25QQP6ELK4XBPUG4NLW4MGXEEYR57VL3UMA, seg_pos={0 0}, upload=58731]: Done uploading piece. err: ecclient: upload failed (node:12L2DTyJJ3v7KRjA424PVwQn9mpeq5Pq4ZTCerhrStgqgc3cVFk, address:104.36.82.52:4089): protocol: write tcp 188.166.225.170:55648->104.36.82.52:4089: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:51:50.428441569 (61h16m0.151004609s) [noise=true , address=104.36.82.56:4233 , piece_id=BHIQILP3H4M337UWEIACAEEIMCFYZCRK7RGMSQ6WF3JXOJJR6X7Q, seg_pos={0 0}, upload=58731]: Done uploading piece. err: ecclient: upload failed (node:1DmQeoQWTnFhQeJFseJoNYzFyo7vZjYKwDtQVJ6pomksWExsTb, address:104.36.82.56:4233): protocol: rpc: dial tcp 104.36.82.56:4233: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:51:50.428447921 (61h16m0.151010961s) [noise=true , address=104.36.82.9:4024 , piece_id=QBP2OLQJAL2XHIYQOLDI3AWNZ5K5O6R7FZOD4O2UZY7VI3HEU7HA, seg_pos={0 0}, upload=58731]: Done uploading piece. err: ecclient: upload failed (node:1bHv9jsVASv3dP3iowvir74PDEHkaaqA3sstiLk7qLSpK2Duzz, address:104.36.82.9:4024): protocol: rpc: dial tcp 104.36.82.9:4024: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:51:50.428470254 (61h16m0.151033294s) [seg_pos={0 0}, upload=58731]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 27 00:51:50.661590409 (61h16m0.384153449s) [upload=58731]: Done uploading segments... Jul 27 00:51:50.661617441 (61h16m0.384180481s) [upload=58731]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.235247519s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [X...............XX......X...XXX..XXX..X..X.X..............X.] duration: 12.219123275s FAIL (2025-07-27 00:51:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.219135057s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 1.827921681s FAIL (2025-07-27 00:51:34 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.827950357s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [.X...............XX......X...XXX..XXX..X..X.X..............X] duration: 1m0.000604671s FAIL (2025-07-27 00:51:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000634919s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 37 out of 60 is failed (threshold=3) [XXXX...XX.XX.X.XXXXXXXXXX.X.XX..XXXXX.XXXX..XXXX.....X....X.] duration: 5.829403678s FAIL (2025-07-27 00:51:22 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.829427292s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X.XX...XXXX.....X..........X..X..X...XX.....................] duration: 7.772097433s FAIL (2025-07-27 00:51:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.772117591s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [.XXXX..X..XX....XX..XXXXX.X.XXXXXXXXX.XXXXXXXX..X.X.X.......] duration: 7.141342593s FAIL (2025-07-27 00:51:13 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== seg_pos={0 0}, upload=58618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:51940->104.36.82.8:4969: use of closed network connection; context canceled Jul 27 00:44:02.225675835 (61h8m11.948238875s) [seg_pos={0 0}, upload=58618]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:44:02.225744527 (61h8m11.948307567s) [upload=58618]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:44:02.225749156 (61h8m11.948312196s) [upload=58618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:44:02.509059970 (61h8m12.23162301s) [upload=58618]: Done uploading segments... Jul 27 00:44:02.509077874 (61h8m12.231640914s) [upload=58618]: Upload finished. err: <nil> Jul 27 00:44:39.669648204 (61h8m49.392211244s) [upload=58630]: Starting upload Jul 27 00:44:39.669713028 (61h8m49.392276068s) [upload=58630]: Done starting upload Jul 27 00:44:39.669834260 (61h8m49.39239729s) [upload=58630]: Uploading segments... Jul 27 00:44:39.669948438 (61h8m49.392511478s) [upload=58630]: Got next segment. Inline: false Jul 27 00:44:39.669993214 (61h8m49.392556254s) [upload=58630]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:44:39.669998714 (61h8m49.392561754s) [upload=58630]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:44:39.906918409 (61h8m49.629481439s) [seg_pos={0 0}, upload=58630]: Begin upload segment... Jul 27 00:44:39.906959377 (61h8m49.629522407s) [seg_pos={0 0}, upload=58630]: Done begin upload segment. Jul 27 00:44:39.906991630 (61h8m49.62955467s) [noise=true , address=45.91.116.4:4937 , piece_id=NLYBLCU3K4VLCWFKGPLNGHTONV6CCX7Y5K3MTPM3GRHLKLAGM7GA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907011507 (61h8m49.629574547s) [noise=true , address=104.36.82.12:4149 , piece_id=NXBQCY5TGXWZZDM4GYHE5HVNUFLAEU7KTA3IQF2QPDCKWF5T276Q, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907355766 (61h8m49.629918806s) [noise=true , address=45.91.116.14:4190 , piece_id=5BMBSQOU4VDTAOKYJ2EHAY55BGBWOZZUQPDFQD47HW2DF2ORLGKA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907550038 (61h8m49.630113078s) [noise=true , address=45.91.116.11:4119 , piece_id=ZTCS3JFUSDI5OS4LE3CVB3RTH6MNDF6LX5TQVC7KG7G6NZDOY75A, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907614301 (61h8m49.630177341s) [noise=true , address=45.91.116.8:4045 , piece_id=NVKHAKHTS6IUW3L7KBEONIW7U6X3YNYX5PVVDNF6GMFDN4D6QWGQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.907923572 (61h8m49.630486612s) [noise=true , address=77.90.31.15:4699 , piece_id=5DNLQI7KVQA6LSMSHN7BX267GCC3W2LMYWPXUH3VKOFMWACFSJUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908056056 (61h8m49.630619096s) [noise=true , address=104.36.82.4:4803 , piece_id=MNNOC3HEF7C3ENJPGK4P6Y6YAOCROIKFILJ7C5U63K4MWDHECMXA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908189431 (61h8m49.630752471s) [noise=true , address=45.91.116.12:4227 , piece_id=HIO2GNSEEYKB7TJLECYDO4ORKH4BNJFBYXDHJDXKOZSWOATJ4HAA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908418639 (61h8m49.630981679s) [noise=true , address=45.91.116.5:4297 , piece_id=S2HC4O2W5W6SU2R6A7RXEQXDPYS74J4FHIBWNB4DTNBKPQZYTVUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908627068 (61h8m49.631190098s) [noise=true , address=104.36.82.8:4969 , piece_id=57SZMUIZBISAZAB6FAUXHI4TY6DGLZIGRGG2KBG26VBT4CR5CFUQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.908839696 (61h8m49.631402736s) [noise=true , address=45.91.116.8:4048 , piece_id=P35BZEGEAWHOGTRF6J7GATKTW6RBMUOT56J5DFX4YRIQVJUGJPJA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909044928 (61h8m49.631607968s) [noise=true , address=104.36.82.9:4006 , piece_id=SZSXMIL2N7P7DNGZW6MSKK6SFIVWUTQADU6RNI3AAGVZYMOJ2EKA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909265650 (61h8m49.63182869s) [noise=true , address=104.36.82.12:4150 , piece_id=QZTCPCAXF2LRJ7QIDWRK35CZEZ3VTZ2S6Y7UWPJ2UMNHW2IPHQZQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909493406 (61h8m49.632056446s) [noise=true , address=104.36.82.6:4875 , piece_id=W5L23TO2POIULWAZR77REJMF5ELCRXHIDDKDLWS6THTDLT2ZAQZQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.909692708 (61h8m49.632255748s) [noise=true , address=45.91.116.10:4081 , piece_id=H7ECBWZIOD3X4MDZIREJIYXVGBVLF45RXXJQ6OJR63IMSA7MO4XQ, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:39.910010295 (61h8m49.632573335s) [noise=true , address=77.90.31.14:4661 , piece_id=MNQA3JE764QDX4DVALBLMHVBHDJNDR53KRRQO3EM26MCNCIY77EA, seg_pos={0 0}, upload=58630]: Uploading piece... Jul 27 00:44:41.604649775 (61h8m51.327212805s) [upload=58630]: Next returned nil segment Jul 27 00:44:41.604666106 (61h8m51.327229146s) [upload=58630]: Waiting for error group managing segments... Jul 27 00:44:41.940055171 (61h8m51.662618211s) [noise=true , address=45.91.116.5:4297 , piece_id=S2HC4O2W5W6SU2R6A7RXEQXDPYS74J4FHIBWNB4DTNBKPQZYTVUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.976252637 (61h8m51.698815677s) [noise=true , address=45.91.116.8:4045 , piece_id=NVKHAKHTS6IUW3L7KBEONIW7U6X3YNYX5PVVDNF6GMFDN4D6QWGQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.977419500 (61h8m51.69998254s) [noise=true , address=104.36.82.4:4803 , piece_id=MNNOC3HEF7C3ENJPGK4P6Y6YAOCROIKFILJ7C5U63K4MWDHECMXA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:41.992292375 (61h8m51.714855455s) [noise=true , address=45.91.116.12:4227 , piece_id=HIO2GNSEEYKB7TJLECYDO4ORKH4BNJFBYXDHJDXKOZSWOATJ4HAA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.005049984 (61h8m51.727613024s) [noise=true , address=45.91.116.10:4081 , piece_id=H7ECBWZIOD3X4MDZIREJIYXVGBVLF45RXXJQ6OJR63IMSA7MO4XQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.007733369 (61h8m51.730296409s) [noise=true , address=45.91.116.4:4937 , piece_id=NLYBLCU3K4VLCWFKGPLNGHTONV6CCX7Y5K3MTPM3GRHLKLAGM7GA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.015889453 (61h8m51.738452483s) [noise=true , address=77.90.31.14:4661 , piece_id=MNQA3JE764QDX4DVALBLMHVBHDJNDR53KRRQO3EM26MCNCIY77EA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.026646625 (61h8m51.749209665s) [noise=true , address=45.91.116.8:4048 , piece_id=P35BZEGEAWHOGTRF6J7GATKTW6RBMUOT56J5DFX4YRIQVJUGJPJA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.032403049 (61h8m51.754966089s) [noise=true , address=45.91.116.14:4190 , piece_id=5BMBSQOU4VDTAOKYJ2EHAY55BGBWOZZUQPDFQD47HW2DF2ORLGKA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.088809390 (61h8m51.81137243s) [noise=true , address=45.91.116.11:4119 , piece_id=ZTCS3JFUSDI5OS4LE3CVB3RTH6MNDF6LX5TQVC7KG7G6NZDOY75A, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.102686731 (61h8m51.825249771s) [noise=true , address=104.36.82.6:4875 , piece_id=W5L23TO2POIULWAZR77REJMF5ELCRXHIDDKDLWS6THTDLT2ZAQZQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:44:42.125894665 (61h8m51.848457695s) [noise=true , address=104.36.82.8:4969 , piece_id=57SZMUIZBISAZAB6FAUXHI4TY6DGLZIGRGG2KBG26VBT4CR5CFUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: <nil> Jul 27 00:45:09.023578831 (61h9m18.746141871s) [noise=true , address=104.36.82.9:4006 , piece_id=SZSXMIL2N7P7DNGZW6MSKK6SFIVWUTQADU6RNI3AAGVZYMOJ2EKA, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:38976->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023652292 (61h9m18.746215332s) [noise=true , address=104.36.82.12:4150 , piece_id=QZTCPCAXF2LRJ7QIDWRK35CZEZ3VTZ2S6Y7UWPJ2UMNHW2IPHQZQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:36462->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023664465 (61h9m18.746227495s) [noise=true , address=104.36.82.12:4149 , piece_id=NXBQCY5TGXWZZDM4GYHE5HVNUFLAEU7KTA3IQF2QPDCKWF5T276Q, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:53774->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023738908 (61h9m18.746301938s) [noise=true , address=77.90.31.15:4699 , piece_id=5DNLQI7KVQA6LSMSHN7BX267GCC3W2LMYWPXUH3VKOFMWACFSJUQ, seg_pos={0 0}, upload=58630]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:47968->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:45:09.023761581 (61h9m18.746324611s) [seg_pos={0 0}, upload=58630]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:45:09.265509153 (61h9m18.988072193s) [upload=58630]: Done uploading segments... Jul 27 00:45:09.265533339 (61h9m18.988096369s) [upload=58630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.14136748s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 38 out of 60 is failed (threshold=3) [XXXXX...XX.XX.X.XXXXXXXXXX.X.XX..XXXXX.XXXX..XXXX.....X....X] duration: 30.001764997s FAIL (2025-07-27 00:50:56 UTC) orbiter warning: uplink: context deadline exceeded Duration: 30.001785465s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 53 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......X] duration: 1m0.231624609s FAIL (2025-07-27 00:50:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== PC3KUDRG7ZT6FKBZGC5D7B4NYDPT6L44GFPVNR6LA, seg_pos={0 0}, upload=58711]: Uploading piece... Jul 27 00:49:44.818337162 (61h13m54.540900202s) [noise=true , address=104.36.82.46:4484 , piece_id=XDZ6SDX6JD7X2USPUG7ZVWRVVP7YZCYHHIYH6ZWP62R4D6CMWHJA, seg_pos={0 0}, upload=58711]: Uploading piece... Jul 27 00:49:44.818529791 (61h13m54.541092871s) [noise=true , address=104.36.82.53:4122 , piece_id=4YHRYCHR3ABUNE3MV3XYLZLJWTZH5XNHAXCTD5KI474UJPWD42EQ, seg_pos={0 0}, upload=58711]: Uploading piece... Jul 27 00:49:44.818719173 (61h13m54.541282203s) [noise=true , address=77.90.31.12:4616 , piece_id=RBC4OKAWJTF6US6EYJPPQLN3AFEGXXMR6XTWVH3QV55PFVJ4GKIQ, seg_pos={0 0}, upload=58711]: Uploading piece... Jul 27 00:49:44.818894979 (61h13m54.541458059s) [noise=true , address=104.36.82.10:4098 , piece_id=GKWC2YCHJYXSQEG52TRWPZ3NEJ42SLVNZISHPZFKEQQQPKHFDWWA, seg_pos={0 0}, upload=58711]: Uploading piece... Jul 27 00:49:44.819033093 (61h13m54.541596133s) [noise=true , address=79.127.205.246:4272 , piece_id=MOWGEDFWP7LBKA23OWQXJDPRWB5QTI24I7XGYSDKOL5KEI6ASVXA, seg_pos={0 0}, upload=58711]: Uploading piece... Jul 27 00:49:44.819331494 (61h13m54.541894535s) [noise=true , address=104.36.82.55:4195 , piece_id=DYURY6RDTAEZVGOB3HC2ILJN7W7TGYG4PMRK4DI6FQIZEIEVXE6A, seg_pos={0 0}, upload=58711]: Uploading piece... Jul 27 00:49:44.819491341 (61h13m54.542054401s) [noise=true , address=77.90.31.10:4538 , piece_id=IPV6PUXIU424KJHUAQ6YPSYN4VQMQQVNKNG6DVTLMLDMLE7CVZPQ, seg_pos={0 0}, upload=58711]: Uploading piece... Jul 27 00:49:44.819623504 (61h13m54.542186534s) [noise=true , address=104.36.82.15:4261 , piece_id=LX753JAFLTI7VULGEEBNL7CT2AJN3XDMHRBLW73WYY3MFAKBAFQA, seg_pos={0 0}, upload=58711]: Uploading piece... Jul 27 00:49:44.819899121 (61h13m54.542462161s) [noise=true , address=104.36.82.44:4870 , piece_id=3W4SOPA5MXJLEP3WL3JSR6EMT2IUFJF6U3T2PG74UKX4HDCPUVOQ, seg_pos={0 0}, upload=58711]: Uploading piece... Jul 27 00:49:44.820050991 (61h13m54.542614041s) [noise=true , address=64.49.13.26:4953 , piece_id=S4U6QL622MET27Y35U66BVLH44NMXRED42SRK7TN7YWA7CGTSOPQ, seg_pos={0 0}, upload=58711]: Uploading piece... Jul 27 00:49:44.820100526 (61h13m54.542663566s) [noise=true , address=79.127.220.101:4041 , piece_id=YDAFNA3MDJTW7TEFVIYOZ7DWDNPEQTOY4TIGV37HYNBXSDGZVWVA, seg_pos={0 0}, upload=58711]: Uploading piece... Jul 27 00:49:44.820416991 (61h13m54.542980031s) [noise=true , address=79.127.201.215:4357 , piece_id=5V5FNXLSPP7G2ASDCPOLWG2I2HEUUFFAV5XYGAP6A3N4B7NTQD5Q, seg_pos={0 0}, upload=58711]: Uploading piece... Jul 27 00:49:44.820534657 (61h13m54.543097697s) [noise=true , address=104.36.82.56:4236 , piece_id=OOL5XMNL6LC56HMZIQCLTJYIYPPGKAVAFJHO47B3PB5NHG55TAIA, seg_pos={0 0}, upload=58711]: Uploading piece... Jul 27 00:49:44.820816917 (61h13m54.543379957s) [noise=true , address=79.127.205.234:4610 , piece_id=DM5G6UJHKGAP27LCRGF4VVH3ZTSOTQN5RFVWN6AOG3GSWCG5MPCQ, seg_pos={0 0}, upload=58711]: Uploading piece... Jul 27 00:49:44.821046626 (61h13m54.543609656s) [noise=true , address=64.49.13.140:4508 , piece_id=QDR7C3IZ4B24U4E26VLMEFZCIPYNZTGUMVXF3NYSWTD6WVTLSGIQ, seg_pos={0 0}, upload=58711]: Uploading piece... Jul 27 00:49:44.821238563 (61h13m54.543801603s) [noise=true , address=79.127.205.235:4686 , piece_id=TYVVTYZOXP6XD57EPV6PFKN6UZZHP3Z2POF5VVWZ5COWTOB2PMJA, seg_pos={0 0}, upload=58711]: Uploading piece... Jul 27 00:49:45.927908929 (61h13m55.650471969s) [upload=58711]: Next returned nil segment Jul 27 00:49:45.927939257 (61h13m55.650502287s) [upload=58711]: Waiting for error group managing segments... Jul 27 00:49:46.208572520 (61h13m55.93113556s) [noise=true , address=79.127.205.246:4272 , piece_id=MOWGEDFWP7LBKA23OWQXJDPRWB5QTI24I7XGYSDKOL5KEI6ASVXA, seg_pos={0 0}, upload=58711]: Done uploading piece. err: <nil> Jul 27 00:49:46.265099722 (61h13m55.987662762s) [noise=true , address=79.127.205.235:4686 , piece_id=TYVVTYZOXP6XD57EPV6PFKN6UZZHP3Z2POF5VVWZ5COWTOB2PMJA, seg_pos={0 0}, upload=58711]: Done uploading piece. err: <nil> Jul 27 00:49:46.292751568 (61h13m56.015314618s) [noise=true , address=64.49.13.140:4508 , piece_id=QDR7C3IZ4B24U4E26VLMEFZCIPYNZTGUMVXF3NYSWTD6WVTLSGIQ, seg_pos={0 0}, upload=58711]: Done uploading piece. err: <nil> Jul 27 00:49:46.364343835 (61h13m56.086906875s) [noise=true , address=79.127.205.234:4610 , piece_id=DM5G6UJHKGAP27LCRGF4VVH3ZTSOTQN5RFVWN6AOG3GSWCG5MPCQ, seg_pos={0 0}, upload=58711]: Done uploading piece. err: <nil> Jul 27 00:49:46.833386096 (61h13m56.555949136s) [noise=true , address=77.90.31.10:4538 , piece_id=IPV6PUXIU424KJHUAQ6YPSYN4VQMQQVNKNG6DVTLMLDMLE7CVZPQ, seg_pos={0 0}, upload=58711]: Done uploading piece. err: <nil> Jul 27 00:49:48.114392688 (61h13m57.836955718s) [noise=true , address=79.127.219.38:4006 , piece_id=QF6GBAPY3JKPC3KUDRG7ZT6FKBZGC5D7B4NYDPT6L44GFPVNR6LA, seg_pos={0 0}, upload=58711]: Done uploading piece. err: <nil> Jul 27 00:49:48.962346493 (61h13m58.684909533s) [noise=true , address=64.49.13.26:4953 , piece_id=S4U6QL622MET27Y35U66BVLH44NMXRED42SRK7TN7YWA7CGTSOPQ, seg_pos={0 0}, upload=58711]: Done uploading piece. err: <nil> Jul 27 00:49:51.849929052 (61h14m1.572492092s) [noise=true , address=79.127.201.215:4357 , piece_id=5V5FNXLSPP7G2ASDCPOLWG2I2HEUUFFAV5XYGAP6A3N4B7NTQD5Q, seg_pos={0 0}, upload=58711]: Done uploading piece. err: <nil> Jul 27 00:49:53.400937102 (61h14m3.123500132s) [noise=true , address=79.127.220.101:4041 , piece_id=YDAFNA3MDJTW7TEFVIYOZ7DWDNPEQTOY4TIGV37HYNBXSDGZVWVA, seg_pos={0 0}, upload=58711]: Done uploading piece. err: <nil> Jul 27 00:50:35.867238194 (61h14m45.589801234s) [noise=true , address=77.90.31.12:4616 , piece_id=RBC4OKAWJTF6US6EYJPPQLN3AFEGXXMR6XTWVH3QV55PFVJ4GKIQ, seg_pos={0 0}, upload=58711]: Done uploading piece. err: <nil> Jul 27 00:50:43.729047504 (61h14m53.451610544s) [noise=true , address=104.36.82.47:4528 , piece_id=HKB6OPO5FX5WE7FPKGRQYDWYINDQ6KFQHKZHEUZ2G5VRIW63AH5Q, seg_pos={0 0}, upload=58711]: Done uploading piece. err: ecclient: upload failed (node:12ArcCrcHdpNgBMfTcPdtTXb65RgnTh8N32ihHAx2RrddxnRYgT, address:104.36.82.47:4528): protocol: write tcp 188.166.225.170:51138->104.36.82.47:4528: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:50:43.729092090 (61h14m53.45165513s) [noise=true , address=104.36.82.56:4236 , piece_id=OOL5XMNL6LC56HMZIQCLTJYIYPPGKAVAFJHO47B3PB5NHG55TAIA, seg_pos={0 0}, upload=58711]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: write tcp 188.166.225.170:45292->104.36.82.56:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:50:43.729130212 (61h14m53.451693242s) [noise=true , address=104.36.82.10:4098 , piece_id=GKWC2YCHJYXSQEG52TRWPZ3NEJ42SLVNZISHPZFKEQQQPKHFDWWA, seg_pos={0 0}, upload=58711]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: write tcp 188.166.225.170:46986->104.36.82.10:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:50:43.729162654 (61h14m53.451725694s) [noise=true , address=104.36.82.44:4870 , piece_id=3W4SOPA5MXJLEP3WL3JSR6EMT2IUFJF6U3T2PG74UKX4HDCPUVOQ, seg_pos={0 0}, upload=58711]: Done uploading piece. err: ecclient: upload failed (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt, address:104.36.82.44:4870): protocol: write tcp 188.166.225.170:49128->104.36.82.44:4870: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:50:43.729376173 (61h14m53.451939203s) [noise=true , address=104.36.82.53:4122 , piece_id=4YHRYCHR3ABUNE3MV3XYLZLJWTZH5XNHAXCTD5KI474UJPWD42EQ, seg_pos={0 0}, upload=58711]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:104.36.82.53:4122): protocol: write tcp 188.166.225.170:54408->104.36.82.53:4122: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:50:43.729416320 (61h14m53.45197935s) [noise=true , address=104.36.82.46:4484 , piece_id=XDZ6SDX6JD7X2USPUG7ZVWRVVP7YZCYHHIYH6ZWP62R4D6CMWHJA, seg_pos={0 0}, upload=58711]: Done uploading piece. err: ecclient: upload failed (node:1a5S5Ar7jNbK9MESghbWfmEgBPNSMN2tEnZeeG1kKzeX2dnSX3, address:104.36.82.46:4484): protocol: write tcp 188.166.225.170:53820->104.36.82.46:4484: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:50:43.729497295 (61h14m53.452060335s) [noise=true , address=104.36.82.55:4195 , piece_id=DYURY6RDTAEZVGOB3HC2ILJN7W7TGYG4PMRK4DI6FQIZEIEVXE6A, seg_pos={0 0}, upload=58711]: Done uploading piece. err: ecclient: upload failed (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, address:104.36.82.55:4195): protocol: write tcp 188.166.225.170:48420->104.36.82.55:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:50:43.729524237 (61h14m53.452087277s) [noise=true , address=104.36.82.15:4261 , piece_id=LX753JAFLTI7VULGEEBNL7CT2AJN3XDMHRBLW73WYY3MFAKBAFQA, seg_pos={0 0}, upload=58711]: Done uploading piece. err: ecclient: upload failed (node:1i1jcJdEP7Ch9uaGcjDaVxpodx55dJEHUKFbcr1GeKQ9waWjYz, address:104.36.82.15:4261): protocol: write tcp 188.166.225.170:53398->104.36.82.15:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:50:43.729547210 (61h14m53.45211025s) [seg_pos={0 0}, upload=58711]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 27 00:50:43.958012209 (61h14m53.680575249s) [upload=58711]: Done uploading segments... Jul 27 00:50:43.958036065 (61h14m53.680599095s) [upload=58711]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.231646801s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X.XX...XXXX.....X..........X..X..X...XX....................] duration: 7.546264225s FAIL (2025-07-27 00:50:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.546284623s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 1.870036953s FAIL (2025-07-27 00:50:25 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.870066378s core uplink us1 vivint

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)