Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........X...X..............................................] duration: 6.50238723s FAIL (2025-08-19 17:22:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.502413952s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.82863318s FAIL (2025-08-19 17:22:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 263h18m33.75518017s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324344006 (263h18m33.755424127s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324633220 (263h18m33.755713351s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324851286 (263h18m33.755931417s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325068942 (263h18m33.756149063s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325283613 (263h18m33.756363744s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325499857 (263h18m33.756579998s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325759794 (263h18m33.756839915s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326059428 (263h18m33.757139549s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326277565 (263h18m33.757357696s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326470835 (263h18m33.757550966s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326673753 (263h18m33.757753884s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326867664 (263h18m33.757947795s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:40.851885389 (263h18m35.28296552s) [upload=273764]: Next returned nil segment Aug 19 16:44:40.851903334 (263h18m35.282983465s) [upload=273764]: Waiting for error group managing segments... Aug 19 16:44:41.244668372 (263h18m35.675748503s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.277280365 (263h18m35.708360496s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.328561665 (263h18m35.759641786s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.331747139 (263h18m35.76282727s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.357720320 (263h18m35.788800481s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.378190805 (263h18m35.809270936s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.486480333 (263h18m35.917560464s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:45:38.230570787 (263h19m32.661650918s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: expected piece hash; context deadline exceeded; EOF Aug 19 16:45:38.230979869 (263h19m32.66206s) [noise=true , address=77.90.31.18:4840 , piece_id=JWOETSFSZ3HELLZNLEASYNGW5OR64TMQVOFYYJAJVTHWCBHVTSLQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: read tcp 188.166.225.170:39946->77.90.31.18:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231017812 (263h19m32.662097943s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 188.166.225.170:44164->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231063489 (263h19m32.66214362s) [noise=true , address=45.91.116.14:4203 , piece_id=WRLBES3FI5EA5AQLZXJYGDROP3KBG3E2F22R4YD6A5KKI44PLV2Q, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 188.166.225.170:51738->45.91.116.14:4203: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231090952 (263h19m32.662171083s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: read tcp 188.166.225.170:57158->77.90.31.15:4707: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231110719 (263h19m32.66219085s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:50784->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231140165 (263h19m32.662220296s) [noise=true , address=77.90.31.16:4749 , piece_id=XCPXVBYLQH345FPIW4KDNLBFUCDDEW2H4B25WIXGOO5FVMAIVKPQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: read tcp 188.166.225.170:51668->77.90.31.16:4749: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231144814 (263h19m32.662224945s) [noise=true , address=77.90.31.19:4865 , piece_id=B3HUFIJZKVZJH4OBBO2DD7S3I7HX4LO2IQ74XJPO4DTW4QER35NA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1bguBghQonV5sTHaWteyUEnpp5KNzBKbvBpbvrUNFDCveHN77X, address:77.90.31.19:4865): protocol: read tcp 188.166.225.170:39398->77.90.31.19:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231177167 (263h19m32.662257298s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: read tcp 188.166.225.170:58526->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231184571 (263h19m32.662264702s) [noise=true , address=45.91.116.12:4236 , piece_id=LCRQS2GOSHOSLZBGQTGB3Q3PA4VN7KXNW66TDZ5O424TPJWTQE3A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH, address:45.91.116.12:4236): protocol: read tcp 188.166.225.170:57166->45.91.116.12:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231189770 (263h19m32.662269901s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 188.166.225.170:54242->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231215850 (263h19m32.662295981s) [seg_pos={0 0}, upload=273764]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 19 16:45:38.459100173 (263h19m32.890180304s) [upload=273764]: Done uploading segments... Aug 19 16:45:38.459128316 (263h19m32.890208447s) [upload=273764]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.828658679s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 1.848206236s FAIL (2025-08-19 17:22:19 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 17:04:06.343826144 (263h38m0.774906265s) [upload=274088]: Starting upload Aug 19 17:04:06.343875278 (263h38m0.774955409s) [upload=274088]: Done starting upload Aug 19 17:04:06.343971703 (263h38m0.775051834s) [upload=274088]: Uploading segments... Aug 19 17:04:06.344028321 (263h38m0.775108453s) [upload=274088]: Got next segment. Inline: true Aug 19 17:04:06.344033120 (263h38m0.775113252s) [upload=274088]: Waiting for error group managing segments... Aug 19 17:04:06.344068187 (263h38m0.775148318s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:04:06.344071964 (263h38m0.775152095s) [upload=274088]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:04:06.344074499 (263h38m0.77515462s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:04:06.653287654 (263h38m1.084367785s) [upload=274088]: Done uploading segments... Aug 19 17:04:06.653300519 (263h38m1.08438065s) [upload=274088]: Upload finished. err: <nil> Duration:* 1.84823386s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [........X......X.XX........................X.............X..] duration: 5.979790851s FAIL (2025-08-19 17:22:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Aug 19 17:20:20.539160574 (263h54m14.970240715s) [upload=274365]: Starting upload Aug 19 17:20:20.539244044 (263h54m14.970324185s) [upload=274365]: Done starting upload Aug 19 17:20:20.539331611 (263h54m14.970411742s) [upload=274365]: Uploading segments... Aug 19 17:20:20.539485776 (263h54m14.970565907s) [upload=274365]: Got next segment. Inline: false Aug 19 17:20:20.539523448 (263h54m14.970603579s) [upload=274365]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:20:20.539528167 (263h54m14.970608288s) [upload=274365]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:20:20.994171861 (263h54m15.425251992s) [seg_pos={0 0}, upload=274365]: Begin upload segment... Aug 19 17:20:20.994220554 (263h54m15.425300685s) [seg_pos={0 0}, upload=274365]: Done begin upload segment. Aug 19 17:20:20.994231936 (263h54m15.425312067s) [noise=true , address=45.91.116.13:4178 , piece_id=I5DDTCHBURRBEAO3PDLND5QP3QA2KFIY63BCDVJZMVG6GL6AMKEQ, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:20.994242185 (263h54m15.425322316s) [noise=true , address=104.36.82.62:4442 , piece_id=4LZX5UFB7ZBCKBHZNGZ4GOLBLB54P4VGNZJ4AVKBEEIV6C37F3AA, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:20.994594229 (263h54m15.42567436s) [noise=true , address=77.90.31.14:4690 , piece_id=NWU5UYKNAPUVX6NNALLDH76HSEY6NY6U2HYBZA2TEORO5GFCOFMQ, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:20.994601091 (263h54m15.425681223s) [noise=true , address=104.36.82.14:4233 , piece_id=FFDWNJCYB3YSBRCAUDBZKNILVB7VIEHBDWEGV4RZYCJN5F4OKN5A, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:20.994866099 (263h54m15.42594622s) [noise=true , address=104.36.82.5:4865 , piece_id=FFLLQPFSE7E3XBUEGJJIDPXEDF3PNVMOLJSS7AYV557TSIE2N27Q, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:20.995618799 (263h54m15.42669893s) [noise=true , address=45.91.116.12:4235 , piece_id=E6FT7UVAYW7KGPPRNGQHMTEJTDJIEJAFUIUVCTCFQDJ3RHMEBC2A, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:20.996295193 (263h54m15.427375324s) [noise=true , address=45.91.116.6:4992 , piece_id=QELWUSUTTHR2CQZQCWK6K5GWSRWJT36N55YOKJM2I2S2BTQNQXXA, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:20.996984071 (263h54m15.428064202s) [noise=true , address=104.36.82.38:4693 , piece_id=QUKUZHYBXLMV4KSEUIYOI6QZSEBQMJ6LLTGWHVZ33WN2P5WMD5NA, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:20.997573749 (263h54m15.42865388s) [noise=true , address=77.90.31.17:4799 , piece_id=4IYAMNEN4ZCFVNYZMI733PWTH4FZPIPAQR6XHC2WTSQC5JMGGNKA, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:20.997965880 (263h54m15.429046011s) [noise=true , address=45.91.116.8:4072 , piece_id=MNLUD2YF3GJQFHZDNZ3OHY2ANUZ6TWYK2E2QDTYBCGC4Q3S6HZKQ, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:20.998227921 (263h54m15.429308052s) [noise=true , address=104.36.82.17:4341 , piece_id=55XGZWBTMCMHTRWQGCFPO56MH6N33YVZGQZSAVZE727OZQKWVBPQ, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:20.998780378 (263h54m15.429860509s) [noise=true , address=104.36.82.23:4464 , piece_id=JOXQ2KEFPBSCEIKSAGK5Y7D3RHMV4NNOKSVQJJXNFXJC6KTM3GKA, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:20.999000198 (263h54m15.430080329s) [noise=true , address=77.90.31.16:4743 , piece_id=UCQ3WJSN27HUEGQGDN2GV5SAMQX5R2OISDND47Q6KVZISMJG5QNQ, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:20.999205442 (263h54m15.430285573s) [noise=true , address=79.127.205.233:4630 , piece_id=TAYE5FTCCR2S4EBZFBB5BOQVRIFZAUGYE6F5OD4LASUFZWA7OM5Q, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:20.999488723 (263h54m15.430568854s) [noise=true , address=104.36.82.18:4388 , piece_id=P5AY5W7GZTL7GMIVGLISZ37RS72XVCR3E7II3AG5NRNHBRNHAXFQ, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:21.000202379 (263h54m15.43128252s) [noise=true , address=104.36.82.36:4632 , piece_id=RKGOZHA4QIXBDEIUAFOJLEIAGYVKWZOOMF6WVQU7HBGFLIKGBV5A, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:21.000522151 (263h54m15.431602282s) [noise=true , address=77.90.31.21:4889 , piece_id=2LH7NOPRA2ENN2PDBCDBKAFLX2AXTNZYF5PI4ZZZ6R3ULMDP3L6Q, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:21.000780805 (263h54m15.431860926s) [noise=true , address=45.91.116.11:4133 , piece_id=QYK6CCA553NY33QAYTSVJJGMOXLZE2I6JFMD2Z442UXWQBYVYCKA, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:22.130965732 (263h54m16.562045853s) [upload=274365]: Next returned nil segment Aug 19 17:20:22.130981351 (263h54m16.562061482s) [upload=274365]: Waiting for error group managing segments... Aug 19 17:20:22.426870888 (263h54m16.857951019s) [noise=true , address=79.127.205.233:4630 , piece_id=TAYE5FTCCR2S4EBZFBB5BOQVRIFZAUGYE6F5OD4LASUFZWA7OM5Q, seg_pos={0 0}, upload=274365]: Done uploading piece. err: <nil> Aug 19 17:20:22.906016242 (263h54m17.337096363s) [noise=true , address=104.36.82.17:4341 , piece_id=55XGZWBTMCMHTRWQGCFPO56MH6N33YVZGQZSAVZE727OZQKWVBPQ, seg_pos={0 0}, upload=274365]: Done uploading piece. err: <nil> Aug 19 17:20:22.939972032 (263h54m17.371052163s) [noise=true , address=104.36.82.62:4442 , piece_id=4LZX5UFB7ZBCKBHZNGZ4GOLBLB54P4VGNZJ4AVKBEEIV6C37F3AA, seg_pos={0 0}, upload=274365]: Done uploading piece. err: <nil> Aug 19 17:20:22.980630166 (263h54m17.411710297s) [noise=true , address=104.36.82.23:4464 , piece_id=JOXQ2KEFPBSCEIKSAGK5Y7D3RHMV4NNOKSVQJJXNFXJC6KTM3GKA, seg_pos={0 0}, upload=274365]: Done uploading piece. err: <nil> Aug 19 17:20:23.003858651 (263h54m17.434938782s) [noise=true , address=104.36.82.36:4632 , piece_id=RKGOZHA4QIXBDEIUAFOJLEIAGYVKWZOOMF6WVQU7HBGFLIKGBV5A, seg_pos={0 0}, upload=274365]: Done uploading piece. err: <nil> Aug 19 17:20:23.028282242 (263h54m17.459362373s) [noise=true , address=104.36.82.5:4865 , piece_id=FFLLQPFSE7E3XBUEGJJIDPXEDF3PNVMOLJSS7AYV557TSIE2N27Q, seg_pos={0 0}, upload=274365]: Done uploading piece. err: <nil> Aug 19 17:20:23.043126465 (263h54m17.474206596s) [noise=true , address=104.36.82.18:4388 , piece_id=P5AY5W7GZTL7GMIVGLISZ37RS72XVCR3E7II3AG5NRNHBRNHAXFQ, seg_pos={0 0}, upload=274365]: Done uploading piece. err: <nil> Aug 19 17:20:23.069146632 (263h54m17.500226763s) [noise=true , address=104.36.82.38:4693 , piece_id=QUKUZHYBXLMV4KSEUIYOI6QZSEBQMJ6LLTGWHVZ33WN2P5WMD5NA, seg_pos={0 0}, upload=274365]: Done uploading piece. err: <nil> Aug 19 17:20:23.153907159 (263h54m17.58498728s) [noise=true , address=104.36.82.14:4233 , piece_id=FFDWNJCYB3YSBRCAUDBZKNILVB7VIEHBDWEGV4RZYCJN5F4OKN5A, seg_pos={0 0}, upload=274365]: Done uploading piece. err: <nil> Aug 19 17:21:15.666636076 (263h55m10.097716207s) [noise=true , address=77.90.31.16:4743 , piece_id=UCQ3WJSN27HUEGQGDN2GV5SAMQX5R2OISDND47Q6KVZISMJG5QNQ, seg_pos={0 0}, upload=274365]: Done uploading piece. err: <nil> Aug 19 17:21:18.678328494 (263h55m13.109408625s) [noise=true , address=45.91.116.6:4992 , piece_id=QELWUSUTTHR2CQZQCWK6K5GWSRWJT36N55YOKJM2I2S2BTQNQXXA, seg_pos={0 0}, upload=274365]: Done uploading piece. err: <nil> Aug 19 17:21:19.891756283 (263h55m14.322836424s) [noise=true , address=77.90.31.14:4690 , piece_id=NWU5UYKNAPUVX6NNALLDH76HSEY6NY6U2HYBZA2TEORO5GFCOFMQ, seg_pos={0 0}, upload=274365]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: expected piece hash; context deadline exceeded; EOF Aug 19 17:21:19.892411977 (263h55m14.323492108s) [noise=true , address=45.91.116.13:4178 , piece_id=I5DDTCHBURRBEAO3PDLND5QP3QA2KFIY63BCDVJZMVG6GL6AMKEQ, seg_pos={0 0}, upload=274365]: Done uploading piece. err: ecclient: upload failed (node:125m3sbLAzXdKnjK9WZmasPccheZX63eDagUXqxVCPiakA9ApNX, address:45.91.116.13:4178): protocol: rpc: dial tcp 45.91.116.13:4178: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:19.892455271 (263h55m14.323535402s) [noise=true , address=45.91.116.8:4072 , piece_id=MNLUD2YF3GJQFHZDNZ3OHY2ANUZ6TWYK2E2QDTYBCGC4Q3S6HZKQ, seg_pos={0 0}, upload=274365]: Done uploading piece. err: ecclient: upload failed (node:1toupLRE5KrBcmqoVvKng6HZeybkwSzzrzwzqH3ZNukQEuEZWu, address:45.91.116.8:4072): protocol: rpc: dial tcp 45.91.116.8:4072: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:19.892487432 (263h55m14.323567553s) [noise=true , address=45.91.116.12:4235 , piece_id=E6FT7UVAYW7KGPPRNGQHMTEJTDJIEJAFUIUVCTCFQDJ3RHMEBC2A, seg_pos={0 0}, upload=274365]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: rpc: dial tcp 45.91.116.12:4235: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:19.892516938 (263h55m14.323597069s) [noise=true , address=77.90.31.17:4799 , piece_id=4IYAMNEN4ZCFVNYZMI733PWTH4FZPIPAQR6XHC2WTSQC5JMGGNKA, seg_pos={0 0}, upload=274365]: Done uploading piece. err: ecclient: upload failed (node:12UhAksgXZuFGAVheaYzP7D4kwVyAjUAq3BtrvdSxszjNCFCTWT, address:77.90.31.17:4799): protocol: rpc: dial tcp 77.90.31.17:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:19.892548779 (263h55m14.32362891s) [noise=true , address=77.90.31.21:4889 , piece_id=2LH7NOPRA2ENN2PDBCDBKAFLX2AXTNZYF5PI4ZZZ6R3ULMDP3L6Q, seg_pos={0 0}, upload=274365]: Done uploading piece. err: ecclient: upload failed (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6, address:77.90.31.21:4889): protocol: read tcp 188.166.225.170:60782->77.90.31.21:4889: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:19.892584287 (263h55m14.323664418s) [noise=true , address=45.91.116.11:4133 , piece_id=QYK6CCA553NY33QAYTSVJJGMOXLZE2I6JFMD2Z442UXWQBYVYCKA, seg_pos={0 0}, upload=274365]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 188.166.225.170:38858->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:19.892598334 (263h55m14.323678465s) [seg_pos={0 0}, upload=274365]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Aug 19 17:21:20.657220121 (263h55m15.088300252s) [upload=274365]: Done uploading segments... Aug 19 17:21:20.657247393 (263h55m15.088327524s) [upload=274365]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.979822363s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X....X........................................X..] duration: 6.06258058s FAIL (2025-08-19 17:22:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.062599657s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........X...X.............................................] duration: 6.570767891s FAIL (2025-08-19 17:22:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.570795394s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X....X........................................X.] duration: 5.978617857s FAIL (2025-08-19 17:21:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.978641183s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 10.683541535s FAIL (2025-08-19 17:21:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 263h18m33.75518017s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324344006 (263h18m33.755424127s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324633220 (263h18m33.755713351s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324851286 (263h18m33.755931417s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325068942 (263h18m33.756149063s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325283613 (263h18m33.756363744s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325499857 (263h18m33.756579998s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325759794 (263h18m33.756839915s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326059428 (263h18m33.757139549s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326277565 (263h18m33.757357696s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326470835 (263h18m33.757550966s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326673753 (263h18m33.757753884s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326867664 (263h18m33.757947795s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:40.851885389 (263h18m35.28296552s) [upload=273764]: Next returned nil segment Aug 19 16:44:40.851903334 (263h18m35.282983465s) [upload=273764]: Waiting for error group managing segments... Aug 19 16:44:41.244668372 (263h18m35.675748503s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.277280365 (263h18m35.708360496s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.328561665 (263h18m35.759641786s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.331747139 (263h18m35.76282727s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.357720320 (263h18m35.788800481s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.378190805 (263h18m35.809270936s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.486480333 (263h18m35.917560464s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:45:38.230570787 (263h19m32.661650918s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: expected piece hash; context deadline exceeded; EOF Aug 19 16:45:38.230979869 (263h19m32.66206s) [noise=true , address=77.90.31.18:4840 , piece_id=JWOETSFSZ3HELLZNLEASYNGW5OR64TMQVOFYYJAJVTHWCBHVTSLQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: read tcp 188.166.225.170:39946->77.90.31.18:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231017812 (263h19m32.662097943s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 188.166.225.170:44164->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231063489 (263h19m32.66214362s) [noise=true , address=45.91.116.14:4203 , piece_id=WRLBES3FI5EA5AQLZXJYGDROP3KBG3E2F22R4YD6A5KKI44PLV2Q, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 188.166.225.170:51738->45.91.116.14:4203: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231090952 (263h19m32.662171083s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: read tcp 188.166.225.170:57158->77.90.31.15:4707: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231110719 (263h19m32.66219085s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:50784->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231140165 (263h19m32.662220296s) [noise=true , address=77.90.31.16:4749 , piece_id=XCPXVBYLQH345FPIW4KDNLBFUCDDEW2H4B25WIXGOO5FVMAIVKPQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: read tcp 188.166.225.170:51668->77.90.31.16:4749: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231144814 (263h19m32.662224945s) [noise=true , address=77.90.31.19:4865 , piece_id=B3HUFIJZKVZJH4OBBO2DD7S3I7HX4LO2IQ74XJPO4DTW4QER35NA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1bguBghQonV5sTHaWteyUEnpp5KNzBKbvBpbvrUNFDCveHN77X, address:77.90.31.19:4865): protocol: read tcp 188.166.225.170:39398->77.90.31.19:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231177167 (263h19m32.662257298s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: read tcp 188.166.225.170:58526->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231184571 (263h19m32.662264702s) [noise=true , address=45.91.116.12:4236 , piece_id=LCRQS2GOSHOSLZBGQTGB3Q3PA4VN7KXNW66TDZ5O424TPJWTQE3A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH, address:45.91.116.12:4236): protocol: read tcp 188.166.225.170:57166->45.91.116.12:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231189770 (263h19m32.662269901s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 188.166.225.170:54242->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231215850 (263h19m32.662295981s) [seg_pos={0 0}, upload=273764]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 19 16:45:38.459100173 (263h19m32.890180304s) [upload=273764]: Done uploading segments... Aug 19 16:45:38.459128316 (263h19m32.890208447s) [upload=273764]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 10.683569859s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........X....XX........................................X.] duration: 6.08190991s FAIL (2025-08-19 17:21:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== h item: pb.BatchRequestItem_ObjectBegin Aug 19 17:20:33.951855883 (263h54m28.382936014s) [upload=274369]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:20:34.187220345 (263h54m28.618300476s) [seg_pos={0 0}, upload=274369]: Begin upload segment... Aug 19 17:20:34.187269749 (263h54m28.61834988s) [seg_pos={0 0}, upload=274369]: Done begin upload segment. Aug 19 17:20:34.187291701 (263h54m28.618371832s) [noise=true , address=104.36.82.4:4803 , piece_id=4HVLZ6KBLW3ZOTWBHYW4PHFXQTMQOR3DUVZIZ4CYJUZABZOFYFZA, seg_pos={0 0}, upload=274369]: Uploading piece... Aug 19 17:20:34.187297161 (263h54m28.618377292s) [noise=true , address=45.91.116.5:4297 , piece_id=R2MZYTHZFJOTGQOJCMBJY4OLEQI6AU2B5HWFXXFPNKQNX4EA5A3Q, seg_pos={0 0}, upload=274369]: Uploading piece... Aug 19 17:20:34.187700463 (263h54m28.618780594s) [noise=true , address=77.90.31.14:4662 , piece_id=6JZWXVJ4JNJ26L7M7YY3HZJ42S5XHVBHYT4LFLFHY6OC43T5K2CQ, seg_pos={0 0}, upload=274369]: Uploading piece... Aug 19 17:20:34.187709811 (263h54m28.618789942s) [noise=true , address=104.36.82.15:4259 , piece_id=VRAMKTGQJRU6LBUCZJMLL22SXAH66PS7RRVFYAFGPIQGMXBMQBIA, seg_pos={0 0}, upload=274369]: Uploading piece... Aug 19 17:20:34.188063417 (263h54m28.619143609s) [noise=true , address=45.91.116.8:4048 , piece_id=DQFBHYAPURKN5LMZBHUQHPSDVBAQJ2GWZICEEVQI5OY2UUQOIA4Q, seg_pos={0 0}, upload=274369]: Uploading piece... Aug 19 17:20:34.188103044 (263h54m28.619183175s) [noise=true , address=45.91.116.7:4010 , piece_id=NJUWVZJL25QJICODPQFSNQ5SYRGKLHWZDWRLLIRP4WM4V5FHXYZQ, seg_pos={0 0}, upload=274369]: Uploading piece... Aug 19 17:20:34.188425561 (263h54m28.619505692s) [noise=true , address=104.36.82.5:4838 , piece_id=QICLKYGS52RSV2NMO7TCWPJKGPQMN4DHNCILIBEWMHWTXRRAZTYQ, seg_pos={0 0}, upload=274369]: Uploading piece... Aug 19 17:20:34.188697190 (263h54m28.619777321s) [noise=true , address=45.91.116.11:4119 , piece_id=SSMI5HBBTRWO2U6BD3TUMWRHONWYFLBVAI4NAFA5CW4SE56LLVKA, seg_pos={0 0}, upload=274369]: Uploading piece... Aug 19 17:20:34.188892304 (263h54m28.619972465s) [noise=true , address=104.36.82.3:4765 , piece_id=XODVBEB2KU4IYPPIZFCVU4WQNFVNDPBOW3U3BGDQL3PQ24TOQKUA, seg_pos={0 0}, upload=274369]: Uploading piece... Aug 19 17:20:34.188992505 (263h54m28.620072636s) [noise=true , address=77.90.31.10:4519 , piece_id=YMTY5LSXAVZDXNDL7XZQQAGARWR3TGJQZCPC76WBCOWSMJL5YSNA, seg_pos={0 0}, upload=274369]: Uploading piece... Aug 19 17:20:34.189218799 (263h54m28.62029893s) [noise=true , address=104.36.82.3:4767 , piece_id=7WGHCB5FYG7IOG6ERU2VW4PJ7AXVBHOTFCI63H7GWGK4F2QIOTHA, seg_pos={0 0}, upload=274369]: Uploading piece... Aug 19 17:20:34.189426285 (263h54m28.620506416s) [noise=true , address=104.36.82.10:4078 , piece_id=CL2HXUKHEMEQY723LRDHRXY7AZG4H3PXHP4N7U3K73G5RVMA3NJA, seg_pos={0 0}, upload=274369]: Uploading piece... Aug 19 17:20:34.189541275 (263h54m28.620621406s) [noise=true , address=45.91.116.4:4937 , piece_id=2G6ZGM7R7I4WPV4XJMLEJZ3TCPPQP7FYVRAUO656SGJEJPF64BXQ, seg_pos={0 0}, upload=274369]: Uploading piece... Aug 19 17:20:34.189707052 (263h54m28.620787183s) [noise=true , address=45.91.116.14:4189 , piece_id=4VUDCARUKOTP7OCTHCDFMEICWENE6LGJKJJNTZSWMNQBD4PZU6OA, seg_pos={0 0}, upload=274369]: Uploading piece... Aug 19 17:20:34.189956890 (263h54m28.621037031s) [noise=true , address=104.36.82.2:4730 , piece_id=VGYSQWX5PPNV7E5V4G3VNS4L2RSWJTFZAWIC6EMMOEAMWX6YLROA, seg_pos={0 0}, upload=274369]: Uploading piece... Aug 19 17:20:34.190405640 (263h54m28.621485771s) [noise=true , address=45.91.116.6:4973 , piece_id=M3KCQXL2OQR2F5UUUDHU6NHRCLOGCVKDGKX2QUO4C5PMPTN3BKRQ, seg_pos={0 0}, upload=274369]: Uploading piece... Aug 19 17:20:35.953426456 (263h54m30.384506587s) [upload=274369]: Next returned nil segment Aug 19 17:20:35.953446906 (263h54m30.384527037s) [upload=274369]: Waiting for error group managing segments... Aug 19 17:20:36.244224040 (263h54m30.675304171s) [noise=true , address=104.36.82.2:4730 , piece_id=VGYSQWX5PPNV7E5V4G3VNS4L2RSWJTFZAWIC6EMMOEAMWX6YLROA, seg_pos={0 0}, upload=274369]: Done uploading piece. err: <nil> Aug 19 17:20:36.283545056 (263h54m30.714625187s) [noise=true , address=104.36.82.3:4767 , piece_id=7WGHCB5FYG7IOG6ERU2VW4PJ7AXVBHOTFCI63H7GWGK4F2QIOTHA, seg_pos={0 0}, upload=274369]: Done uploading piece. err: <nil> Aug 19 17:20:36.315320805 (263h54m30.746400926s) [noise=true , address=104.36.82.15:4259 , piece_id=VRAMKTGQJRU6LBUCZJMLL22SXAH66PS7RRVFYAFGPIQGMXBMQBIA, seg_pos={0 0}, upload=274369]: Done uploading piece. err: <nil> Aug 19 17:20:36.333653582 (263h54m30.764733713s) [noise=true , address=104.36.82.5:4838 , piece_id=QICLKYGS52RSV2NMO7TCWPJKGPQMN4DHNCILIBEWMHWTXRRAZTYQ, seg_pos={0 0}, upload=274369]: Done uploading piece. err: <nil> Aug 19 17:20:36.341320833 (263h54m30.772400964s) [noise=true , address=104.36.82.3:4765 , piece_id=XODVBEB2KU4IYPPIZFCVU4WQNFVNDPBOW3U3BGDQL3PQ24TOQKUA, seg_pos={0 0}, upload=274369]: Done uploading piece. err: <nil> Aug 19 17:20:36.362985816 (263h54m30.794065947s) [noise=true , address=104.36.82.10:4078 , piece_id=CL2HXUKHEMEQY723LRDHRXY7AZG4H3PXHP4N7U3K73G5RVMA3NJA, seg_pos={0 0}, upload=274369]: Done uploading piece. err: <nil> Aug 19 17:20:36.405842357 (263h54m30.836922488s) [noise=true , address=104.36.82.4:4803 , piece_id=4HVLZ6KBLW3ZOTWBHYW4PHFXQTMQOR3DUVZIZ4CYJUZABZOFYFZA, seg_pos={0 0}, upload=274369]: Done uploading piece. err: <nil> Aug 19 17:21:03.312919779 (263h54m57.743999911s) [noise=true , address=45.91.116.6:4973 , piece_id=M3KCQXL2OQR2F5UUUDHU6NHRCLOGCVKDGKX2QUO4C5PMPTN3BKRQ, seg_pos={0 0}, upload=274369]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:03.312996346 (263h54m57.744076477s) [noise=true , address=77.90.31.14:4662 , piece_id=6JZWXVJ4JNJ26L7M7YY3HZJ42S5XHVBHYT4LFLFHY6OC43T5K2CQ, seg_pos={0 0}, upload=274369]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: rpc: dial tcp 77.90.31.14:4662: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:03.313028257 (263h54m57.744108378s) [noise=true , address=45.91.116.8:4048 , piece_id=DQFBHYAPURKN5LMZBHUQHPSDVBAQJ2GWZICEEVQI5OY2UUQOIA4Q, seg_pos={0 0}, upload=274369]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: rpc: dial tcp 45.91.116.8:4048: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:03.313091788 (263h54m57.744171929s) [noise=true , address=45.91.116.4:4937 , piece_id=2G6ZGM7R7I4WPV4XJMLEJZ3TCPPQP7FYVRAUO656SGJEJPF64BXQ, seg_pos={0 0}, upload=274369]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:03.313310567 (263h54m57.744390688s) [noise=true , address=45.91.116.14:4189 , piece_id=4VUDCARUKOTP7OCTHCDFMEICWENE6LGJKJJNTZSWMNQBD4PZU6OA, seg_pos={0 0}, upload=274369]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: read tcp 188.166.225.170:39560->45.91.116.14:4189: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:03.313346896 (263h54m57.744427027s) [noise=true , address=45.91.116.7:4010 , piece_id=NJUWVZJL25QJICODPQFSNQ5SYRGKLHWZDWRLLIRP4WM4V5FHXYZQ, seg_pos={0 0}, upload=274369]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:40802->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:03.313409927 (263h54m57.744490058s) [noise=true , address=45.91.116.5:4297 , piece_id=R2MZYTHZFJOTGQOJCMBJY4OLEQI6AU2B5HWFXXFPNKQNX4EA5A3Q, seg_pos={0 0}, upload=274369]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: read tcp 188.166.225.170:42698->45.91.116.5:4297: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:03.313496453 (263h54m57.744576594s) [noise=true , address=45.91.116.11:4119 , piece_id=SSMI5HBBTRWO2U6BD3TUMWRHONWYFLBVAI4NAFA5CW4SE56LLVKA, seg_pos={0 0}, upload=274369]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:34674->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:03.313549454 (263h54m57.744629585s) [noise=true , address=77.90.31.10:4519 , piece_id=YMTY5LSXAVZDXNDL7XZQQAGARWR3TGJQZCPC76WBCOWSMJL5YSNA, seg_pos={0 0}, upload=274369]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: read tcp 188.166.225.170:35850->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:03.313570605 (263h54m57.744650736s) [seg_pos={0 0}, upload=274369]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Aug 19 17:21:03.535590631 (263h54m57.966670762s) [upload=274369]: Done uploading segments... Aug 19 17:21:03.535615509 (263h54m57.96669564s) [upload=274369]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.081942962s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........X......X.XX........................X.............X.] duration: 14.823150516s FAIL (2025-08-19 17:21:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Aug 19 17:20:20.539160574 (263h54m14.970240715s) [upload=274365]: Starting upload Aug 19 17:20:20.539244044 (263h54m14.970324185s) [upload=274365]: Done starting upload Aug 19 17:20:20.539331611 (263h54m14.970411742s) [upload=274365]: Uploading segments... Aug 19 17:20:20.539485776 (263h54m14.970565907s) [upload=274365]: Got next segment. Inline: false Aug 19 17:20:20.539523448 (263h54m14.970603579s) [upload=274365]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:20:20.539528167 (263h54m14.970608288s) [upload=274365]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:20:20.994171861 (263h54m15.425251992s) [seg_pos={0 0}, upload=274365]: Begin upload segment... Aug 19 17:20:20.994220554 (263h54m15.425300685s) [seg_pos={0 0}, upload=274365]: Done begin upload segment. Aug 19 17:20:20.994231936 (263h54m15.425312067s) [noise=true , address=45.91.116.13:4178 , piece_id=I5DDTCHBURRBEAO3PDLND5QP3QA2KFIY63BCDVJZMVG6GL6AMKEQ, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:20.994242185 (263h54m15.425322316s) [noise=true , address=104.36.82.62:4442 , piece_id=4LZX5UFB7ZBCKBHZNGZ4GOLBLB54P4VGNZJ4AVKBEEIV6C37F3AA, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:20.994594229 (263h54m15.42567436s) [noise=true , address=77.90.31.14:4690 , piece_id=NWU5UYKNAPUVX6NNALLDH76HSEY6NY6U2HYBZA2TEORO5GFCOFMQ, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:20.994601091 (263h54m15.425681223s) [noise=true , address=104.36.82.14:4233 , piece_id=FFDWNJCYB3YSBRCAUDBZKNILVB7VIEHBDWEGV4RZYCJN5F4OKN5A, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:20.994866099 (263h54m15.42594622s) [noise=true , address=104.36.82.5:4865 , piece_id=FFLLQPFSE7E3XBUEGJJIDPXEDF3PNVMOLJSS7AYV557TSIE2N27Q, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:20.995618799 (263h54m15.42669893s) [noise=true , address=45.91.116.12:4235 , piece_id=E6FT7UVAYW7KGPPRNGQHMTEJTDJIEJAFUIUVCTCFQDJ3RHMEBC2A, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:20.996295193 (263h54m15.427375324s) [noise=true , address=45.91.116.6:4992 , piece_id=QELWUSUTTHR2CQZQCWK6K5GWSRWJT36N55YOKJM2I2S2BTQNQXXA, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:20.996984071 (263h54m15.428064202s) [noise=true , address=104.36.82.38:4693 , piece_id=QUKUZHYBXLMV4KSEUIYOI6QZSEBQMJ6LLTGWHVZ33WN2P5WMD5NA, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:20.997573749 (263h54m15.42865388s) [noise=true , address=77.90.31.17:4799 , piece_id=4IYAMNEN4ZCFVNYZMI733PWTH4FZPIPAQR6XHC2WTSQC5JMGGNKA, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:20.997965880 (263h54m15.429046011s) [noise=true , address=45.91.116.8:4072 , piece_id=MNLUD2YF3GJQFHZDNZ3OHY2ANUZ6TWYK2E2QDTYBCGC4Q3S6HZKQ, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:20.998227921 (263h54m15.429308052s) [noise=true , address=104.36.82.17:4341 , piece_id=55XGZWBTMCMHTRWQGCFPO56MH6N33YVZGQZSAVZE727OZQKWVBPQ, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:20.998780378 (263h54m15.429860509s) [noise=true , address=104.36.82.23:4464 , piece_id=JOXQ2KEFPBSCEIKSAGK5Y7D3RHMV4NNOKSVQJJXNFXJC6KTM3GKA, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:20.999000198 (263h54m15.430080329s) [noise=true , address=77.90.31.16:4743 , piece_id=UCQ3WJSN27HUEGQGDN2GV5SAMQX5R2OISDND47Q6KVZISMJG5QNQ, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:20.999205442 (263h54m15.430285573s) [noise=true , address=79.127.205.233:4630 , piece_id=TAYE5FTCCR2S4EBZFBB5BOQVRIFZAUGYE6F5OD4LASUFZWA7OM5Q, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:20.999488723 (263h54m15.430568854s) [noise=true , address=104.36.82.18:4388 , piece_id=P5AY5W7GZTL7GMIVGLISZ37RS72XVCR3E7II3AG5NRNHBRNHAXFQ, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:21.000202379 (263h54m15.43128252s) [noise=true , address=104.36.82.36:4632 , piece_id=RKGOZHA4QIXBDEIUAFOJLEIAGYVKWZOOMF6WVQU7HBGFLIKGBV5A, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:21.000522151 (263h54m15.431602282s) [noise=true , address=77.90.31.21:4889 , piece_id=2LH7NOPRA2ENN2PDBCDBKAFLX2AXTNZYF5PI4ZZZ6R3ULMDP3L6Q, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:21.000780805 (263h54m15.431860926s) [noise=true , address=45.91.116.11:4133 , piece_id=QYK6CCA553NY33QAYTSVJJGMOXLZE2I6JFMD2Z442UXWQBYVYCKA, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:22.130965732 (263h54m16.562045853s) [upload=274365]: Next returned nil segment Aug 19 17:20:22.130981351 (263h54m16.562061482s) [upload=274365]: Waiting for error group managing segments... Aug 19 17:20:22.426870888 (263h54m16.857951019s) [noise=true , address=79.127.205.233:4630 , piece_id=TAYE5FTCCR2S4EBZFBB5BOQVRIFZAUGYE6F5OD4LASUFZWA7OM5Q, seg_pos={0 0}, upload=274365]: Done uploading piece. err: <nil> Aug 19 17:20:22.906016242 (263h54m17.337096363s) [noise=true , address=104.36.82.17:4341 , piece_id=55XGZWBTMCMHTRWQGCFPO56MH6N33YVZGQZSAVZE727OZQKWVBPQ, seg_pos={0 0}, upload=274365]: Done uploading piece. err: <nil> Aug 19 17:20:22.939972032 (263h54m17.371052163s) [noise=true , address=104.36.82.62:4442 , piece_id=4LZX5UFB7ZBCKBHZNGZ4GOLBLB54P4VGNZJ4AVKBEEIV6C37F3AA, seg_pos={0 0}, upload=274365]: Done uploading piece. err: <nil> Aug 19 17:20:22.980630166 (263h54m17.411710297s) [noise=true , address=104.36.82.23:4464 , piece_id=JOXQ2KEFPBSCEIKSAGK5Y7D3RHMV4NNOKSVQJJXNFXJC6KTM3GKA, seg_pos={0 0}, upload=274365]: Done uploading piece. err: <nil> Aug 19 17:20:23.003858651 (263h54m17.434938782s) [noise=true , address=104.36.82.36:4632 , piece_id=RKGOZHA4QIXBDEIUAFOJLEIAGYVKWZOOMF6WVQU7HBGFLIKGBV5A, seg_pos={0 0}, upload=274365]: Done uploading piece. err: <nil> Aug 19 17:20:23.028282242 (263h54m17.459362373s) [noise=true , address=104.36.82.5:4865 , piece_id=FFLLQPFSE7E3XBUEGJJIDPXEDF3PNVMOLJSS7AYV557TSIE2N27Q, seg_pos={0 0}, upload=274365]: Done uploading piece. err: <nil> Aug 19 17:20:23.043126465 (263h54m17.474206596s) [noise=true , address=104.36.82.18:4388 , piece_id=P5AY5W7GZTL7GMIVGLISZ37RS72XVCR3E7II3AG5NRNHBRNHAXFQ, seg_pos={0 0}, upload=274365]: Done uploading piece. err: <nil> Aug 19 17:20:23.069146632 (263h54m17.500226763s) [noise=true , address=104.36.82.38:4693 , piece_id=QUKUZHYBXLMV4KSEUIYOI6QZSEBQMJ6LLTGWHVZ33WN2P5WMD5NA, seg_pos={0 0}, upload=274365]: Done uploading piece. err: <nil> Aug 19 17:20:23.153907159 (263h54m17.58498728s) [noise=true , address=104.36.82.14:4233 , piece_id=FFDWNJCYB3YSBRCAUDBZKNILVB7VIEHBDWEGV4RZYCJN5F4OKN5A, seg_pos={0 0}, upload=274365]: Done uploading piece. err: <nil> Aug 19 17:21:15.666636076 (263h55m10.097716207s) [noise=true , address=77.90.31.16:4743 , piece_id=UCQ3WJSN27HUEGQGDN2GV5SAMQX5R2OISDND47Q6KVZISMJG5QNQ, seg_pos={0 0}, upload=274365]: Done uploading piece. err: <nil> Aug 19 17:21:18.678328494 (263h55m13.109408625s) [noise=true , address=45.91.116.6:4992 , piece_id=QELWUSUTTHR2CQZQCWK6K5GWSRWJT36N55YOKJM2I2S2BTQNQXXA, seg_pos={0 0}, upload=274365]: Done uploading piece. err: <nil> Aug 19 17:21:19.891756283 (263h55m14.322836424s) [noise=true , address=77.90.31.14:4690 , piece_id=NWU5UYKNAPUVX6NNALLDH76HSEY6NY6U2HYBZA2TEORO5GFCOFMQ, seg_pos={0 0}, upload=274365]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: expected piece hash; context deadline exceeded; EOF Aug 19 17:21:19.892411977 (263h55m14.323492108s) [noise=true , address=45.91.116.13:4178 , piece_id=I5DDTCHBURRBEAO3PDLND5QP3QA2KFIY63BCDVJZMVG6GL6AMKEQ, seg_pos={0 0}, upload=274365]: Done uploading piece. err: ecclient: upload failed (node:125m3sbLAzXdKnjK9WZmasPccheZX63eDagUXqxVCPiakA9ApNX, address:45.91.116.13:4178): protocol: rpc: dial tcp 45.91.116.13:4178: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:19.892455271 (263h55m14.323535402s) [noise=true , address=45.91.116.8:4072 , piece_id=MNLUD2YF3GJQFHZDNZ3OHY2ANUZ6TWYK2E2QDTYBCGC4Q3S6HZKQ, seg_pos={0 0}, upload=274365]: Done uploading piece. err: ecclient: upload failed (node:1toupLRE5KrBcmqoVvKng6HZeybkwSzzrzwzqH3ZNukQEuEZWu, address:45.91.116.8:4072): protocol: rpc: dial tcp 45.91.116.8:4072: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:19.892487432 (263h55m14.323567553s) [noise=true , address=45.91.116.12:4235 , piece_id=E6FT7UVAYW7KGPPRNGQHMTEJTDJIEJAFUIUVCTCFQDJ3RHMEBC2A, seg_pos={0 0}, upload=274365]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: rpc: dial tcp 45.91.116.12:4235: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:19.892516938 (263h55m14.323597069s) [noise=true , address=77.90.31.17:4799 , piece_id=4IYAMNEN4ZCFVNYZMI733PWTH4FZPIPAQR6XHC2WTSQC5JMGGNKA, seg_pos={0 0}, upload=274365]: Done uploading piece. err: ecclient: upload failed (node:12UhAksgXZuFGAVheaYzP7D4kwVyAjUAq3BtrvdSxszjNCFCTWT, address:77.90.31.17:4799): protocol: rpc: dial tcp 77.90.31.17:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:19.892548779 (263h55m14.32362891s) [noise=true , address=77.90.31.21:4889 , piece_id=2LH7NOPRA2ENN2PDBCDBKAFLX2AXTNZYF5PI4ZZZ6R3ULMDP3L6Q, seg_pos={0 0}, upload=274365]: Done uploading piece. err: ecclient: upload failed (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6, address:77.90.31.21:4889): protocol: read tcp 188.166.225.170:60782->77.90.31.21:4889: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:19.892584287 (263h55m14.323664418s) [noise=true , address=45.91.116.11:4133 , piece_id=QYK6CCA553NY33QAYTSVJJGMOXLZE2I6JFMD2Z442UXWQBYVYCKA, seg_pos={0 0}, upload=274365]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 188.166.225.170:38858->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:19.892598334 (263h55m14.323678465s) [seg_pos={0 0}, upload=274365]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Aug 19 17:21:20.657220121 (263h55m15.088300252s) [upload=274365]: Done uploading segments... Aug 19 17:21:20.657247393 (263h55m15.088327524s) [upload=274365]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 14.823171176s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........X...X............................................] duration: 10.683338861s FAIL (2025-08-19 17:21:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.683350472s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 3.254641852s FAIL (2025-08-19 17:21:25 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 17:04:06.343826144 (263h38m0.774906265s) [upload=274088]: Starting upload Aug 19 17:04:06.343875278 (263h38m0.774955409s) [upload=274088]: Done starting upload Aug 19 17:04:06.343971703 (263h38m0.775051834s) [upload=274088]: Uploading segments... Aug 19 17:04:06.344028321 (263h38m0.775108453s) [upload=274088]: Got next segment. Inline: true Aug 19 17:04:06.344033120 (263h38m0.775113252s) [upload=274088]: Waiting for error group managing segments... Aug 19 17:04:06.344068187 (263h38m0.775148318s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:04:06.344071964 (263h38m0.775152095s) [upload=274088]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:04:06.344074499 (263h38m0.77515462s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:04:06.653287654 (263h38m1.084367785s) [upload=274088]: Done uploading segments... Aug 19 17:04:06.653300519 (263h38m1.08438065s) [upload=274088]: Upload finished. err: <nil> Duration:* 3.254674656s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........X......X.XX........................X.............X] duration: 1m0.767538021s FAIL (2025-08-19 17:21:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Aug 19 17:20:20.539160574 (263h54m14.970240715s) [upload=274365]: Starting upload Aug 19 17:20:20.539244044 (263h54m14.970324185s) [upload=274365]: Done starting upload Aug 19 17:20:20.539331611 (263h54m14.970411742s) [upload=274365]: Uploading segments... Aug 19 17:20:20.539485776 (263h54m14.970565907s) [upload=274365]: Got next segment. Inline: false Aug 19 17:20:20.539523448 (263h54m14.970603579s) [upload=274365]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:20:20.539528167 (263h54m14.970608288s) [upload=274365]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:20:20.994171861 (263h54m15.425251992s) [seg_pos={0 0}, upload=274365]: Begin upload segment... Aug 19 17:20:20.994220554 (263h54m15.425300685s) [seg_pos={0 0}, upload=274365]: Done begin upload segment. Aug 19 17:20:20.994231936 (263h54m15.425312067s) [noise=true , address=45.91.116.13:4178 , piece_id=I5DDTCHBURRBEAO3PDLND5QP3QA2KFIY63BCDVJZMVG6GL6AMKEQ, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:20.994242185 (263h54m15.425322316s) [noise=true , address=104.36.82.62:4442 , piece_id=4LZX5UFB7ZBCKBHZNGZ4GOLBLB54P4VGNZJ4AVKBEEIV6C37F3AA, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:20.994594229 (263h54m15.42567436s) [noise=true , address=77.90.31.14:4690 , piece_id=NWU5UYKNAPUVX6NNALLDH76HSEY6NY6U2HYBZA2TEORO5GFCOFMQ, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:20.994601091 (263h54m15.425681223s) [noise=true , address=104.36.82.14:4233 , piece_id=FFDWNJCYB3YSBRCAUDBZKNILVB7VIEHBDWEGV4RZYCJN5F4OKN5A, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:20.994866099 (263h54m15.42594622s) [noise=true , address=104.36.82.5:4865 , piece_id=FFLLQPFSE7E3XBUEGJJIDPXEDF3PNVMOLJSS7AYV557TSIE2N27Q, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:20.995618799 (263h54m15.42669893s) [noise=true , address=45.91.116.12:4235 , piece_id=E6FT7UVAYW7KGPPRNGQHMTEJTDJIEJAFUIUVCTCFQDJ3RHMEBC2A, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:20.996295193 (263h54m15.427375324s) [noise=true , address=45.91.116.6:4992 , piece_id=QELWUSUTTHR2CQZQCWK6K5GWSRWJT36N55YOKJM2I2S2BTQNQXXA, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:20.996984071 (263h54m15.428064202s) [noise=true , address=104.36.82.38:4693 , piece_id=QUKUZHYBXLMV4KSEUIYOI6QZSEBQMJ6LLTGWHVZ33WN2P5WMD5NA, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:20.997573749 (263h54m15.42865388s) [noise=true , address=77.90.31.17:4799 , piece_id=4IYAMNEN4ZCFVNYZMI733PWTH4FZPIPAQR6XHC2WTSQC5JMGGNKA, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:20.997965880 (263h54m15.429046011s) [noise=true , address=45.91.116.8:4072 , piece_id=MNLUD2YF3GJQFHZDNZ3OHY2ANUZ6TWYK2E2QDTYBCGC4Q3S6HZKQ, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:20.998227921 (263h54m15.429308052s) [noise=true , address=104.36.82.17:4341 , piece_id=55XGZWBTMCMHTRWQGCFPO56MH6N33YVZGQZSAVZE727OZQKWVBPQ, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:20.998780378 (263h54m15.429860509s) [noise=true , address=104.36.82.23:4464 , piece_id=JOXQ2KEFPBSCEIKSAGK5Y7D3RHMV4NNOKSVQJJXNFXJC6KTM3GKA, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:20.999000198 (263h54m15.430080329s) [noise=true , address=77.90.31.16:4743 , piece_id=UCQ3WJSN27HUEGQGDN2GV5SAMQX5R2OISDND47Q6KVZISMJG5QNQ, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:20.999205442 (263h54m15.430285573s) [noise=true , address=79.127.205.233:4630 , piece_id=TAYE5FTCCR2S4EBZFBB5BOQVRIFZAUGYE6F5OD4LASUFZWA7OM5Q, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:20.999488723 (263h54m15.430568854s) [noise=true , address=104.36.82.18:4388 , piece_id=P5AY5W7GZTL7GMIVGLISZ37RS72XVCR3E7II3AG5NRNHBRNHAXFQ, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:21.000202379 (263h54m15.43128252s) [noise=true , address=104.36.82.36:4632 , piece_id=RKGOZHA4QIXBDEIUAFOJLEIAGYVKWZOOMF6WVQU7HBGFLIKGBV5A, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:21.000522151 (263h54m15.431602282s) [noise=true , address=77.90.31.21:4889 , piece_id=2LH7NOPRA2ENN2PDBCDBKAFLX2AXTNZYF5PI4ZZZ6R3ULMDP3L6Q, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:21.000780805 (263h54m15.431860926s) [noise=true , address=45.91.116.11:4133 , piece_id=QYK6CCA553NY33QAYTSVJJGMOXLZE2I6JFMD2Z442UXWQBYVYCKA, seg_pos={0 0}, upload=274365]: Uploading piece... Aug 19 17:20:22.130965732 (263h54m16.562045853s) [upload=274365]: Next returned nil segment Aug 19 17:20:22.130981351 (263h54m16.562061482s) [upload=274365]: Waiting for error group managing segments... Aug 19 17:20:22.426870888 (263h54m16.857951019s) [noise=true , address=79.127.205.233:4630 , piece_id=TAYE5FTCCR2S4EBZFBB5BOQVRIFZAUGYE6F5OD4LASUFZWA7OM5Q, seg_pos={0 0}, upload=274365]: Done uploading piece. err: <nil> Aug 19 17:20:22.906016242 (263h54m17.337096363s) [noise=true , address=104.36.82.17:4341 , piece_id=55XGZWBTMCMHTRWQGCFPO56MH6N33YVZGQZSAVZE727OZQKWVBPQ, seg_pos={0 0}, upload=274365]: Done uploading piece. err: <nil> Aug 19 17:20:22.939972032 (263h54m17.371052163s) [noise=true , address=104.36.82.62:4442 , piece_id=4LZX5UFB7ZBCKBHZNGZ4GOLBLB54P4VGNZJ4AVKBEEIV6C37F3AA, seg_pos={0 0}, upload=274365]: Done uploading piece. err: <nil> Aug 19 17:20:22.980630166 (263h54m17.411710297s) [noise=true , address=104.36.82.23:4464 , piece_id=JOXQ2KEFPBSCEIKSAGK5Y7D3RHMV4NNOKSVQJJXNFXJC6KTM3GKA, seg_pos={0 0}, upload=274365]: Done uploading piece. err: <nil> Aug 19 17:20:23.003858651 (263h54m17.434938782s) [noise=true , address=104.36.82.36:4632 , piece_id=RKGOZHA4QIXBDEIUAFOJLEIAGYVKWZOOMF6WVQU7HBGFLIKGBV5A, seg_pos={0 0}, upload=274365]: Done uploading piece. err: <nil> Aug 19 17:20:23.028282242 (263h54m17.459362373s) [noise=true , address=104.36.82.5:4865 , piece_id=FFLLQPFSE7E3XBUEGJJIDPXEDF3PNVMOLJSS7AYV557TSIE2N27Q, seg_pos={0 0}, upload=274365]: Done uploading piece. err: <nil> Aug 19 17:20:23.043126465 (263h54m17.474206596s) [noise=true , address=104.36.82.18:4388 , piece_id=P5AY5W7GZTL7GMIVGLISZ37RS72XVCR3E7II3AG5NRNHBRNHAXFQ, seg_pos={0 0}, upload=274365]: Done uploading piece. err: <nil> Aug 19 17:20:23.069146632 (263h54m17.500226763s) [noise=true , address=104.36.82.38:4693 , piece_id=QUKUZHYBXLMV4KSEUIYOI6QZSEBQMJ6LLTGWHVZ33WN2P5WMD5NA, seg_pos={0 0}, upload=274365]: Done uploading piece. err: <nil> Aug 19 17:20:23.153907159 (263h54m17.58498728s) [noise=true , address=104.36.82.14:4233 , piece_id=FFDWNJCYB3YSBRCAUDBZKNILVB7VIEHBDWEGV4RZYCJN5F4OKN5A, seg_pos={0 0}, upload=274365]: Done uploading piece. err: <nil> Aug 19 17:21:15.666636076 (263h55m10.097716207s) [noise=true , address=77.90.31.16:4743 , piece_id=UCQ3WJSN27HUEGQGDN2GV5SAMQX5R2OISDND47Q6KVZISMJG5QNQ, seg_pos={0 0}, upload=274365]: Done uploading piece. err: <nil> Aug 19 17:21:18.678328494 (263h55m13.109408625s) [noise=true , address=45.91.116.6:4992 , piece_id=QELWUSUTTHR2CQZQCWK6K5GWSRWJT36N55YOKJM2I2S2BTQNQXXA, seg_pos={0 0}, upload=274365]: Done uploading piece. err: <nil> Aug 19 17:21:19.891756283 (263h55m14.322836424s) [noise=true , address=77.90.31.14:4690 , piece_id=NWU5UYKNAPUVX6NNALLDH76HSEY6NY6U2HYBZA2TEORO5GFCOFMQ, seg_pos={0 0}, upload=274365]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: expected piece hash; context deadline exceeded; EOF Aug 19 17:21:19.892411977 (263h55m14.323492108s) [noise=true , address=45.91.116.13:4178 , piece_id=I5DDTCHBURRBEAO3PDLND5QP3QA2KFIY63BCDVJZMVG6GL6AMKEQ, seg_pos={0 0}, upload=274365]: Done uploading piece. err: ecclient: upload failed (node:125m3sbLAzXdKnjK9WZmasPccheZX63eDagUXqxVCPiakA9ApNX, address:45.91.116.13:4178): protocol: rpc: dial tcp 45.91.116.13:4178: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:19.892455271 (263h55m14.323535402s) [noise=true , address=45.91.116.8:4072 , piece_id=MNLUD2YF3GJQFHZDNZ3OHY2ANUZ6TWYK2E2QDTYBCGC4Q3S6HZKQ, seg_pos={0 0}, upload=274365]: Done uploading piece. err: ecclient: upload failed (node:1toupLRE5KrBcmqoVvKng6HZeybkwSzzrzwzqH3ZNukQEuEZWu, address:45.91.116.8:4072): protocol: rpc: dial tcp 45.91.116.8:4072: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:19.892487432 (263h55m14.323567553s) [noise=true , address=45.91.116.12:4235 , piece_id=E6FT7UVAYW7KGPPRNGQHMTEJTDJIEJAFUIUVCTCFQDJ3RHMEBC2A, seg_pos={0 0}, upload=274365]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: rpc: dial tcp 45.91.116.12:4235: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:19.892516938 (263h55m14.323597069s) [noise=true , address=77.90.31.17:4799 , piece_id=4IYAMNEN4ZCFVNYZMI733PWTH4FZPIPAQR6XHC2WTSQC5JMGGNKA, seg_pos={0 0}, upload=274365]: Done uploading piece. err: ecclient: upload failed (node:12UhAksgXZuFGAVheaYzP7D4kwVyAjUAq3BtrvdSxszjNCFCTWT, address:77.90.31.17:4799): protocol: rpc: dial tcp 77.90.31.17:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:19.892548779 (263h55m14.32362891s) [noise=true , address=77.90.31.21:4889 , piece_id=2LH7NOPRA2ENN2PDBCDBKAFLX2AXTNZYF5PI4ZZZ6R3ULMDP3L6Q, seg_pos={0 0}, upload=274365]: Done uploading piece. err: ecclient: upload failed (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6, address:77.90.31.21:4889): protocol: read tcp 188.166.225.170:60782->77.90.31.21:4889: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:19.892584287 (263h55m14.323664418s) [noise=true , address=45.91.116.11:4133 , piece_id=QYK6CCA553NY33QAYTSVJJGMOXLZE2I6JFMD2Z442UXWQBYVYCKA, seg_pos={0 0}, upload=274365]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 188.166.225.170:38858->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:19.892598334 (263h55m14.323678465s) [seg_pos={0 0}, upload=274365]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Aug 19 17:21:20.657220121 (263h55m15.088300252s) [upload=274365]: Done uploading segments... Aug 19 17:21:20.657247393 (263h55m15.088327524s) [upload=274365]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.76756364s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............X...X...........................................] duration: 40.203730436s FAIL (2025-08-19 17:21:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 40.20375327s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 23.971121511s FAIL (2025-08-19 17:21:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 263h18m33.75518017s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324344006 (263h18m33.755424127s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324633220 (263h18m33.755713351s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324851286 (263h18m33.755931417s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325068942 (263h18m33.756149063s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325283613 (263h18m33.756363744s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325499857 (263h18m33.756579998s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325759794 (263h18m33.756839915s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326059428 (263h18m33.757139549s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326277565 (263h18m33.757357696s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326470835 (263h18m33.757550966s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326673753 (263h18m33.757753884s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326867664 (263h18m33.757947795s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:40.851885389 (263h18m35.28296552s) [upload=273764]: Next returned nil segment Aug 19 16:44:40.851903334 (263h18m35.282983465s) [upload=273764]: Waiting for error group managing segments... Aug 19 16:44:41.244668372 (263h18m35.675748503s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.277280365 (263h18m35.708360496s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.328561665 (263h18m35.759641786s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.331747139 (263h18m35.76282727s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.357720320 (263h18m35.788800481s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.378190805 (263h18m35.809270936s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.486480333 (263h18m35.917560464s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:45:38.230570787 (263h19m32.661650918s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: expected piece hash; context deadline exceeded; EOF Aug 19 16:45:38.230979869 (263h19m32.66206s) [noise=true , address=77.90.31.18:4840 , piece_id=JWOETSFSZ3HELLZNLEASYNGW5OR64TMQVOFYYJAJVTHWCBHVTSLQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: read tcp 188.166.225.170:39946->77.90.31.18:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231017812 (263h19m32.662097943s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 188.166.225.170:44164->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231063489 (263h19m32.66214362s) [noise=true , address=45.91.116.14:4203 , piece_id=WRLBES3FI5EA5AQLZXJYGDROP3KBG3E2F22R4YD6A5KKI44PLV2Q, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 188.166.225.170:51738->45.91.116.14:4203: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231090952 (263h19m32.662171083s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: read tcp 188.166.225.170:57158->77.90.31.15:4707: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231110719 (263h19m32.66219085s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:50784->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231140165 (263h19m32.662220296s) [noise=true , address=77.90.31.16:4749 , piece_id=XCPXVBYLQH345FPIW4KDNLBFUCDDEW2H4B25WIXGOO5FVMAIVKPQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: read tcp 188.166.225.170:51668->77.90.31.16:4749: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231144814 (263h19m32.662224945s) [noise=true , address=77.90.31.19:4865 , piece_id=B3HUFIJZKVZJH4OBBO2DD7S3I7HX4LO2IQ74XJPO4DTW4QER35NA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1bguBghQonV5sTHaWteyUEnpp5KNzBKbvBpbvrUNFDCveHN77X, address:77.90.31.19:4865): protocol: read tcp 188.166.225.170:39398->77.90.31.19:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231177167 (263h19m32.662257298s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: read tcp 188.166.225.170:58526->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231184571 (263h19m32.662264702s) [noise=true , address=45.91.116.12:4236 , piece_id=LCRQS2GOSHOSLZBGQTGB3Q3PA4VN7KXNW66TDZ5O424TPJWTQE3A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH, address:45.91.116.12:4236): protocol: read tcp 188.166.225.170:57166->45.91.116.12:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231189770 (263h19m32.662269901s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 188.166.225.170:54242->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231215850 (263h19m32.662295981s) [seg_pos={0 0}, upload=273764]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 19 16:45:38.459100173 (263h19m32.890180304s) [upload=273764]: Done uploading segments... Aug 19 16:45:38.459128316 (263h19m32.890208447s) [upload=273764]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 23.971140517s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............X....XX........................................X] duration: 30.224962214s FAIL (2025-08-19 17:21:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== h item: pb.BatchRequestItem_ObjectBegin Aug 19 17:20:33.951855883 (263h54m28.382936014s) [upload=274369]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:20:34.187220345 (263h54m28.618300476s) [seg_pos={0 0}, upload=274369]: Begin upload segment... Aug 19 17:20:34.187269749 (263h54m28.61834988s) [seg_pos={0 0}, upload=274369]: Done begin upload segment. Aug 19 17:20:34.187291701 (263h54m28.618371832s) [noise=true , address=104.36.82.4:4803 , piece_id=4HVLZ6KBLW3ZOTWBHYW4PHFXQTMQOR3DUVZIZ4CYJUZABZOFYFZA, seg_pos={0 0}, upload=274369]: Uploading piece... Aug 19 17:20:34.187297161 (263h54m28.618377292s) [noise=true , address=45.91.116.5:4297 , piece_id=R2MZYTHZFJOTGQOJCMBJY4OLEQI6AU2B5HWFXXFPNKQNX4EA5A3Q, seg_pos={0 0}, upload=274369]: Uploading piece... Aug 19 17:20:34.187700463 (263h54m28.618780594s) [noise=true , address=77.90.31.14:4662 , piece_id=6JZWXVJ4JNJ26L7M7YY3HZJ42S5XHVBHYT4LFLFHY6OC43T5K2CQ, seg_pos={0 0}, upload=274369]: Uploading piece... Aug 19 17:20:34.187709811 (263h54m28.618789942s) [noise=true , address=104.36.82.15:4259 , piece_id=VRAMKTGQJRU6LBUCZJMLL22SXAH66PS7RRVFYAFGPIQGMXBMQBIA, seg_pos={0 0}, upload=274369]: Uploading piece... Aug 19 17:20:34.188063417 (263h54m28.619143609s) [noise=true , address=45.91.116.8:4048 , piece_id=DQFBHYAPURKN5LMZBHUQHPSDVBAQJ2GWZICEEVQI5OY2UUQOIA4Q, seg_pos={0 0}, upload=274369]: Uploading piece... Aug 19 17:20:34.188103044 (263h54m28.619183175s) [noise=true , address=45.91.116.7:4010 , piece_id=NJUWVZJL25QJICODPQFSNQ5SYRGKLHWZDWRLLIRP4WM4V5FHXYZQ, seg_pos={0 0}, upload=274369]: Uploading piece... Aug 19 17:20:34.188425561 (263h54m28.619505692s) [noise=true , address=104.36.82.5:4838 , piece_id=QICLKYGS52RSV2NMO7TCWPJKGPQMN4DHNCILIBEWMHWTXRRAZTYQ, seg_pos={0 0}, upload=274369]: Uploading piece... Aug 19 17:20:34.188697190 (263h54m28.619777321s) [noise=true , address=45.91.116.11:4119 , piece_id=SSMI5HBBTRWO2U6BD3TUMWRHONWYFLBVAI4NAFA5CW4SE56LLVKA, seg_pos={0 0}, upload=274369]: Uploading piece... Aug 19 17:20:34.188892304 (263h54m28.619972465s) [noise=true , address=104.36.82.3:4765 , piece_id=XODVBEB2KU4IYPPIZFCVU4WQNFVNDPBOW3U3BGDQL3PQ24TOQKUA, seg_pos={0 0}, upload=274369]: Uploading piece... Aug 19 17:20:34.188992505 (263h54m28.620072636s) [noise=true , address=77.90.31.10:4519 , piece_id=YMTY5LSXAVZDXNDL7XZQQAGARWR3TGJQZCPC76WBCOWSMJL5YSNA, seg_pos={0 0}, upload=274369]: Uploading piece... Aug 19 17:20:34.189218799 (263h54m28.62029893s) [noise=true , address=104.36.82.3:4767 , piece_id=7WGHCB5FYG7IOG6ERU2VW4PJ7AXVBHOTFCI63H7GWGK4F2QIOTHA, seg_pos={0 0}, upload=274369]: Uploading piece... Aug 19 17:20:34.189426285 (263h54m28.620506416s) [noise=true , address=104.36.82.10:4078 , piece_id=CL2HXUKHEMEQY723LRDHRXY7AZG4H3PXHP4N7U3K73G5RVMA3NJA, seg_pos={0 0}, upload=274369]: Uploading piece... Aug 19 17:20:34.189541275 (263h54m28.620621406s) [noise=true , address=45.91.116.4:4937 , piece_id=2G6ZGM7R7I4WPV4XJMLEJZ3TCPPQP7FYVRAUO656SGJEJPF64BXQ, seg_pos={0 0}, upload=274369]: Uploading piece... Aug 19 17:20:34.189707052 (263h54m28.620787183s) [noise=true , address=45.91.116.14:4189 , piece_id=4VUDCARUKOTP7OCTHCDFMEICWENE6LGJKJJNTZSWMNQBD4PZU6OA, seg_pos={0 0}, upload=274369]: Uploading piece... Aug 19 17:20:34.189956890 (263h54m28.621037031s) [noise=true , address=104.36.82.2:4730 , piece_id=VGYSQWX5PPNV7E5V4G3VNS4L2RSWJTFZAWIC6EMMOEAMWX6YLROA, seg_pos={0 0}, upload=274369]: Uploading piece... Aug 19 17:20:34.190405640 (263h54m28.621485771s) [noise=true , address=45.91.116.6:4973 , piece_id=M3KCQXL2OQR2F5UUUDHU6NHRCLOGCVKDGKX2QUO4C5PMPTN3BKRQ, seg_pos={0 0}, upload=274369]: Uploading piece... Aug 19 17:20:35.953426456 (263h54m30.384506587s) [upload=274369]: Next returned nil segment Aug 19 17:20:35.953446906 (263h54m30.384527037s) [upload=274369]: Waiting for error group managing segments... Aug 19 17:20:36.244224040 (263h54m30.675304171s) [noise=true , address=104.36.82.2:4730 , piece_id=VGYSQWX5PPNV7E5V4G3VNS4L2RSWJTFZAWIC6EMMOEAMWX6YLROA, seg_pos={0 0}, upload=274369]: Done uploading piece. err: <nil> Aug 19 17:20:36.283545056 (263h54m30.714625187s) [noise=true , address=104.36.82.3:4767 , piece_id=7WGHCB5FYG7IOG6ERU2VW4PJ7AXVBHOTFCI63H7GWGK4F2QIOTHA, seg_pos={0 0}, upload=274369]: Done uploading piece. err: <nil> Aug 19 17:20:36.315320805 (263h54m30.746400926s) [noise=true , address=104.36.82.15:4259 , piece_id=VRAMKTGQJRU6LBUCZJMLL22SXAH66PS7RRVFYAFGPIQGMXBMQBIA, seg_pos={0 0}, upload=274369]: Done uploading piece. err: <nil> Aug 19 17:20:36.333653582 (263h54m30.764733713s) [noise=true , address=104.36.82.5:4838 , piece_id=QICLKYGS52RSV2NMO7TCWPJKGPQMN4DHNCILIBEWMHWTXRRAZTYQ, seg_pos={0 0}, upload=274369]: Done uploading piece. err: <nil> Aug 19 17:20:36.341320833 (263h54m30.772400964s) [noise=true , address=104.36.82.3:4765 , piece_id=XODVBEB2KU4IYPPIZFCVU4WQNFVNDPBOW3U3BGDQL3PQ24TOQKUA, seg_pos={0 0}, upload=274369]: Done uploading piece. err: <nil> Aug 19 17:20:36.362985816 (263h54m30.794065947s) [noise=true , address=104.36.82.10:4078 , piece_id=CL2HXUKHEMEQY723LRDHRXY7AZG4H3PXHP4N7U3K73G5RVMA3NJA, seg_pos={0 0}, upload=274369]: Done uploading piece. err: <nil> Aug 19 17:20:36.405842357 (263h54m30.836922488s) [noise=true , address=104.36.82.4:4803 , piece_id=4HVLZ6KBLW3ZOTWBHYW4PHFXQTMQOR3DUVZIZ4CYJUZABZOFYFZA, seg_pos={0 0}, upload=274369]: Done uploading piece. err: <nil> Aug 19 17:21:03.312919779 (263h54m57.743999911s) [noise=true , address=45.91.116.6:4973 , piece_id=M3KCQXL2OQR2F5UUUDHU6NHRCLOGCVKDGKX2QUO4C5PMPTN3BKRQ, seg_pos={0 0}, upload=274369]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:03.312996346 (263h54m57.744076477s) [noise=true , address=77.90.31.14:4662 , piece_id=6JZWXVJ4JNJ26L7M7YY3HZJ42S5XHVBHYT4LFLFHY6OC43T5K2CQ, seg_pos={0 0}, upload=274369]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: rpc: dial tcp 77.90.31.14:4662: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:03.313028257 (263h54m57.744108378s) [noise=true , address=45.91.116.8:4048 , piece_id=DQFBHYAPURKN5LMZBHUQHPSDVBAQJ2GWZICEEVQI5OY2UUQOIA4Q, seg_pos={0 0}, upload=274369]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: rpc: dial tcp 45.91.116.8:4048: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:03.313091788 (263h54m57.744171929s) [noise=true , address=45.91.116.4:4937 , piece_id=2G6ZGM7R7I4WPV4XJMLEJZ3TCPPQP7FYVRAUO656SGJEJPF64BXQ, seg_pos={0 0}, upload=274369]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:03.313310567 (263h54m57.744390688s) [noise=true , address=45.91.116.14:4189 , piece_id=4VUDCARUKOTP7OCTHCDFMEICWENE6LGJKJJNTZSWMNQBD4PZU6OA, seg_pos={0 0}, upload=274369]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: read tcp 188.166.225.170:39560->45.91.116.14:4189: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:03.313346896 (263h54m57.744427027s) [noise=true , address=45.91.116.7:4010 , piece_id=NJUWVZJL25QJICODPQFSNQ5SYRGKLHWZDWRLLIRP4WM4V5FHXYZQ, seg_pos={0 0}, upload=274369]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:40802->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:03.313409927 (263h54m57.744490058s) [noise=true , address=45.91.116.5:4297 , piece_id=R2MZYTHZFJOTGQOJCMBJY4OLEQI6AU2B5HWFXXFPNKQNX4EA5A3Q, seg_pos={0 0}, upload=274369]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: read tcp 188.166.225.170:42698->45.91.116.5:4297: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:03.313496453 (263h54m57.744576594s) [noise=true , address=45.91.116.11:4119 , piece_id=SSMI5HBBTRWO2U6BD3TUMWRHONWYFLBVAI4NAFA5CW4SE56LLVKA, seg_pos={0 0}, upload=274369]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:34674->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:03.313549454 (263h54m57.744629585s) [noise=true , address=77.90.31.10:4519 , piece_id=YMTY5LSXAVZDXNDL7XZQQAGARWR3TGJQZCPC76WBCOWSMJL5YSNA, seg_pos={0 0}, upload=274369]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: read tcp 188.166.225.170:35850->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:21:03.313570605 (263h54m57.744650736s) [seg_pos={0 0}, upload=274369]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Aug 19 17:21:03.535590631 (263h54m57.966670762s) [upload=274369]: Done uploading segments... Aug 19 17:21:03.535615509 (263h54m57.96669564s) [upload=274369]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.224984746s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X....X........................................X] duration: 30.000400895s FAIL (2025-08-19 17:21:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000422065s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 3.865034321s FAIL (2025-08-19 17:20:21 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 17:04:06.343826144 (263h38m0.774906265s) [upload=274088]: Starting upload Aug 19 17:04:06.343875278 (263h38m0.774955409s) [upload=274088]: Done starting upload Aug 19 17:04:06.343971703 (263h38m0.775051834s) [upload=274088]: Uploading segments... Aug 19 17:04:06.344028321 (263h38m0.775108453s) [upload=274088]: Got next segment. Inline: true Aug 19 17:04:06.344033120 (263h38m0.775113252s) [upload=274088]: Waiting for error group managing segments... Aug 19 17:04:06.344068187 (263h38m0.775148318s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:04:06.344071964 (263h38m0.775152095s) [upload=274088]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:04:06.344074499 (263h38m0.77515462s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:04:06.653287654 (263h38m1.084367785s) [upload=274088]: Done uploading segments... Aug 19 17:04:06.653300519 (263h38m1.08438065s) [upload=274088]: Upload finished. err: <nil> Duration:* 3.86506419s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............X...X..........................................] duration: 6.696221064s FAIL (2025-08-19 17:20:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.696248056s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X....X........................................] duration: 5.991372959s FAIL (2025-08-19 17:19:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.991397145s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 1.9541673s FAIL (2025-08-19 17:19:55 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 17:04:06.343826144 (263h38m0.774906265s) [upload=274088]: Starting upload Aug 19 17:04:06.343875278 (263h38m0.774955409s) [upload=274088]: Done starting upload Aug 19 17:04:06.343971703 (263h38m0.775051834s) [upload=274088]: Uploading segments... Aug 19 17:04:06.344028321 (263h38m0.775108453s) [upload=274088]: Got next segment. Inline: true Aug 19 17:04:06.344033120 (263h38m0.775113252s) [upload=274088]: Waiting for error group managing segments... Aug 19 17:04:06.344068187 (263h38m0.775148318s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:04:06.344071964 (263h38m0.775152095s) [upload=274088]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:04:06.344074499 (263h38m0.77515462s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:04:06.653287654 (263h38m1.084367785s) [upload=274088]: Done uploading segments... Aug 19 17:04:06.653300519 (263h38m1.08438065s) [upload=274088]: Upload finished. err: <nil> Duration:* 1.954187208s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.561089537s FAIL (2025-08-19 17:19:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 263h18m33.75518017s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324344006 (263h18m33.755424127s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324633220 (263h18m33.755713351s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324851286 (263h18m33.755931417s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325068942 (263h18m33.756149063s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325283613 (263h18m33.756363744s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325499857 (263h18m33.756579998s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325759794 (263h18m33.756839915s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326059428 (263h18m33.757139549s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326277565 (263h18m33.757357696s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326470835 (263h18m33.757550966s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326673753 (263h18m33.757753884s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326867664 (263h18m33.757947795s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:40.851885389 (263h18m35.28296552s) [upload=273764]: Next returned nil segment Aug 19 16:44:40.851903334 (263h18m35.282983465s) [upload=273764]: Waiting for error group managing segments... Aug 19 16:44:41.244668372 (263h18m35.675748503s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.277280365 (263h18m35.708360496s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.328561665 (263h18m35.759641786s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.331747139 (263h18m35.76282727s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.357720320 (263h18m35.788800481s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.378190805 (263h18m35.809270936s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.486480333 (263h18m35.917560464s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:45:38.230570787 (263h19m32.661650918s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: expected piece hash; context deadline exceeded; EOF Aug 19 16:45:38.230979869 (263h19m32.66206s) [noise=true , address=77.90.31.18:4840 , piece_id=JWOETSFSZ3HELLZNLEASYNGW5OR64TMQVOFYYJAJVTHWCBHVTSLQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: read tcp 188.166.225.170:39946->77.90.31.18:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231017812 (263h19m32.662097943s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 188.166.225.170:44164->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231063489 (263h19m32.66214362s) [noise=true , address=45.91.116.14:4203 , piece_id=WRLBES3FI5EA5AQLZXJYGDROP3KBG3E2F22R4YD6A5KKI44PLV2Q, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 188.166.225.170:51738->45.91.116.14:4203: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231090952 (263h19m32.662171083s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: read tcp 188.166.225.170:57158->77.90.31.15:4707: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231110719 (263h19m32.66219085s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:50784->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231140165 (263h19m32.662220296s) [noise=true , address=77.90.31.16:4749 , piece_id=XCPXVBYLQH345FPIW4KDNLBFUCDDEW2H4B25WIXGOO5FVMAIVKPQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: read tcp 188.166.225.170:51668->77.90.31.16:4749: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231144814 (263h19m32.662224945s) [noise=true , address=77.90.31.19:4865 , piece_id=B3HUFIJZKVZJH4OBBO2DD7S3I7HX4LO2IQ74XJPO4DTW4QER35NA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1bguBghQonV5sTHaWteyUEnpp5KNzBKbvBpbvrUNFDCveHN77X, address:77.90.31.19:4865): protocol: read tcp 188.166.225.170:39398->77.90.31.19:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231177167 (263h19m32.662257298s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: read tcp 188.166.225.170:58526->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231184571 (263h19m32.662264702s) [noise=true , address=45.91.116.12:4236 , piece_id=LCRQS2GOSHOSLZBGQTGB3Q3PA4VN7KXNW66TDZ5O424TPJWTQE3A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH, address:45.91.116.12:4236): protocol: read tcp 188.166.225.170:57166->45.91.116.12:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231189770 (263h19m32.662269901s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 188.166.225.170:54242->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231215850 (263h19m32.662295981s) [seg_pos={0 0}, upload=273764]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 19 16:45:38.459100173 (263h19m32.890180304s) [upload=273764]: Done uploading segments... Aug 19 16:45:38.459128316 (263h19m32.890208447s) [upload=273764]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.56111664s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........X......X.XX........................X.............] duration: 5.851201239s FAIL (2025-08-19 17:19:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== Aug 19 17:09:15.684096389 (263h43m10.11517652s) [upload=274176]: Starting upload Aug 19 17:09:15.684143008 (263h43m10.115223139s) [upload=274176]: Done starting upload Aug 19 17:09:15.684256185 (263h43m10.115336306s) [upload=274176]: Uploading segments... Aug 19 17:09:15.684468862 (263h43m10.115548993s) [upload=274176]: Got next segment. Inline: false Aug 19 17:09:15.684504950 (263h43m10.115585082s) [upload=274176]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:09:15.684509770 (263h43m10.115589901s) [upload=274176]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:09:16.135567491 (263h43m10.566647622s) [seg_pos={0 0}, upload=274176]: Begin upload segment... Aug 19 17:09:16.135613589 (263h43m10.56669372s) [seg_pos={0 0}, upload=274176]: Done begin upload segment. Aug 19 17:09:16.135636573 (263h43m10.566716695s) [noise=true , address=104.36.82.5:4853 , piece_id=K6TRSISY75NXYJNAAEALQ7W4RA26P5TGIVRMHX5SGOIF2GJ7X3GQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.135699574 (263h43m10.566779705s) [noise=true , address=104.36.82.60:4341 , piece_id=5NWVHAFJXPCBGPMRT42H4C3LPE3XKBDKPSX44D3DOWF6XXXJ5LFA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136028283 (263h43m10.567108404s) [noise=true , address=79.127.220.98:4432 , piece_id=4VA6KKCGA5GM55N3SD7FNUILD5XWYHE5VFRBFDIBAGVEK2VHQELA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136206604 (263h43m10.567286735s) [noise=true , address=104.36.82.22:4538 , piece_id=YUOOOPSZLQRNVU557APDO5FTOCY4FEYAHCBZQEE6YWMBPK5ZJYNA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136295745 (263h43m10.567375876s) [noise=true , address=77.90.31.11:4564 , piece_id=TX3C4VLEEOJOHBN7VDK4KIROG42PDDCB3SK26FZA7PM2455YZCMA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136640434 (263h43m10.567720565s) [noise=true , address=104.36.82.16:4315 , piece_id=P3SYICVJ5HSVY4MM62FDXQXJARZLIOQYNPBCT2SKYFPYZ3G3MSWQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136649702 (263h43m10.567729833s) [noise=true , address=104.36.82.44:4874 , piece_id=D3L2BM57MTGKIWKRSSEL7DRXUOK7LLKHL7M3Y3WAFCBILVDEV24Q, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136950888 (263h43m10.568031019s) [noise=true , address=104.36.82.35:4605 , piece_id=ORHH35ESP7KPGGDRD7ANECFHW7YTA2VFZTB5EMRCBQX5EWB7ZU4Q, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136961468 (263h43m10.568041599s) [noise=true , address=77.90.31.10:4535 , piece_id=N6LDVXERZAIR3EZ2TAMFQBM5Y565FMSP44I5KAWC32DTIOOM3FIQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137201027 (263h43m10.568281158s) [noise=true , address=77.90.31.12:4611 , piece_id=BANXKRHIMMOQFBSC4FW2E3SMNXCQERG2OTD7GXMM56T3X6JPNQGQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137260470 (263h43m10.568340601s) [noise=true , address=104.36.82.11:4134 , piece_id=BSGC4ADI6LMK3E2YEYPWFMYS3GC5VA42YM22VIJEGZ2JANYMLBYQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137463418 (263h43m10.568543549s) [noise=true , address=45.91.116.10:4110 , piece_id=KCZZOHPLSRNA45VXOMW7AKUYMRRFRUA52KSOBHNDRJPCB3RMV5HA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137553992 (263h43m10.568634123s) [noise=true , address=104.36.82.55:4224 , piece_id=PGZXUHGOONWDPIB4ZHAKLL4F2UWMZJPAL7BOTXIGSQWESLUPWGQA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137682447 (263h43m10.568762578s) [noise=true , address=45.91.116.12:4238 , piece_id=JAUKDCYIMCIOX4EFHQMCNCBLVZEIBKODPBL5QVBPOJIB554KLNRQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137883032 (263h43m10.568963163s) [noise=true , address=77.90.31.16:4743 , piece_id=YKPMLYPRMAZEBX7ORWEICH5AWKVS2J74D25W4ANIJJOUIBX5UBRA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137912418 (263h43m10.568992539s) [noise=true , address=104.36.82.23:4455 , piece_id=L2M5P5LRKVLVVDSYIOBV4SR3EAH2UO2JUZBGKZKIWBUIGFZPVEOA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.138136216 (263h43m10.569216347s) [noise=true , address=104.36.82.7:4930 , piece_id=EGTR52TZDAVTPPRCJLK4GOUXUWW2U7XJMQOV5P3JCKAKOK7GG5JQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.138374362 (263h43m10.569454493s) [noise=true , address=77.90.31.18:4839 , piece_id=BG553GCZSJEH5ORINSIQ7IUYU3UBQ3WO73ZAGRB7W45HOASJMBVQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:17.616571604 (263h43m12.047651765s) [upload=274176]: Next returned nil segment Aug 19 17:09:17.616591412 (263h43m12.047671543s) [upload=274176]: Waiting for error group managing segments... Aug 19 17:09:18.008239751 (263h43m12.439319882s) [noise=true , address=79.127.220.98:4432 , piece_id=4VA6KKCGA5GM55N3SD7FNUILD5XWYHE5VFRBFDIBAGVEK2VHQELA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.039911371 (263h43m12.470991502s) [noise=true , address=45.91.116.12:4238 , piece_id=JAUKDCYIMCIOX4EFHQMCNCBLVZEIBKODPBL5QVBPOJIB554KLNRQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.065695686 (263h43m12.496775817s) [noise=true , address=77.90.31.16:4743 , piece_id=YKPMLYPRMAZEBX7ORWEICH5AWKVS2J74D25W4ANIJJOUIBX5UBRA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.080085439 (263h43m12.51116557s) [noise=true , address=45.91.116.10:4110 , piece_id=KCZZOHPLSRNA45VXOMW7AKUYMRRFRUA52KSOBHNDRJPCB3RMV5HA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.131454157 (263h43m12.562534288s) [noise=true , address=104.36.82.11:4134 , piece_id=BSGC4ADI6LMK3E2YEYPWFMYS3GC5VA42YM22VIJEGZ2JANYMLBYQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.143557214 (263h43m12.574637345s) [noise=true , address=104.36.82.7:4930 , piece_id=EGTR52TZDAVTPPRCJLK4GOUXUWW2U7XJMQOV5P3JCKAKOK7GG5JQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.156135631 (263h43m12.587215762s) [noise=true , address=77.90.31.12:4611 , piece_id=BANXKRHIMMOQFBSC4FW2E3SMNXCQERG2OTD7GXMM56T3X6JPNQGQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.159345252 (263h43m12.590425373s) [noise=true , address=104.36.82.55:4224 , piece_id=PGZXUHGOONWDPIB4ZHAKLL4F2UWMZJPAL7BOTXIGSQWESLUPWGQA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.169106219 (263h43m12.60018635s) [noise=true , address=104.36.82.22:4538 , piece_id=YUOOOPSZLQRNVU557APDO5FTOCY4FEYAHCBZQEE6YWMBPK5ZJYNA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.188273593 (263h43m12.619353724s) [noise=true , address=104.36.82.35:4605 , piece_id=ORHH35ESP7KPGGDRD7ANECFHW7YTA2VFZTB5EMRCBQX5EWB7ZU4Q, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.204131846 (263h43m12.635211977s) [noise=true , address=104.36.82.16:4315 , piece_id=P3SYICVJ5HSVY4MM62FDXQXJARZLIOQYNPBCT2SKYFPYZ3G3MSWQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.217541965 (263h43m12.648622096s) [noise=true , address=104.36.82.60:4341 , piece_id=5NWVHAFJXPCBGPMRT42H4C3LPE3XKBDKPSX44D3DOWF6XXXJ5LFA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.222565827 (263h43m12.653645948s) [noise=true , address=104.36.82.23:4455 , piece_id=L2M5P5LRKVLVVDSYIOBV4SR3EAH2UO2JUZBGKZKIWBUIGFZPVEOA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.377619011 (263h43m12.808699132s) [noise=true , address=104.36.82.44:4874 , piece_id=D3L2BM57MTGKIWKRSSEL7DRXUOK7LLKHL7M3Y3WAFCBILVDEV24Q, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.377646604 (263h43m12.808726725s) [seg_pos={0 0}, upload=274176]: Segment reached optimal threshold of 14 pieces. Aug 19 17:09:18.377754100 (263h43m12.808834231s) [noise=true , address=104.36.82.5:4853 , piece_id=K6TRSISY75NXYJNAAEALQ7W4RA26P5TGIVRMHX5SGOIF2GJ7X3GQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377895570 (263h43m12.808975701s) [noise=true , address=77.90.31.18:4839 , piece_id=BG553GCZSJEH5ORINSIQ7IUYU3UBQ3WO73ZAGRB7W45HOASJMBVQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gmWwyaQ6zM6QdayBFpXdE9VSE1aaQcBHFcukJoASjjHWCDsNx): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377915529 (263h43m12.80899566s) [noise=true , address=77.90.31.11:4564 , piece_id=TX3C4VLEEOJOHBN7VDK4KIROG42PDDCB3SK26FZA7PM2455YZCMA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377974832 (263h43m12.809054963s) [noise=true , address=77.90.31.10:4535 , piece_id=N6LDVXERZAIR3EZ2TAMFQBM5Y565FMSP44I5KAWC32DTIOOM3FIQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377999008 (263h43m12.80907914s) [seg_pos={0 0}, upload=274176]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 17:09:18.378145599 (263h43m12.80922573s) [upload=274176]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 17:09:18.378150488 (263h43m12.809230619s) [upload=274176]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:09:18.983488776 (263h43m13.414568907s) [upload=274176]: Done uploading segments... Aug 19 17:09:18.983507222 (263h43m13.414587353s) [upload=274176]: Upload finished. err: <nil> Duration: 5.851262015s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............X...X.........................................] duration: 6.611355479s FAIL (2025-08-19 17:19:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.611375478s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X....XX........................................] duration: 5.85318767s FAIL (2025-08-19 17:19:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 19 16:49:12.801154313 (263h23m7.232234445s) [upload=273839]: Starting upload Aug 19 16:49:12.801215942 (263h23m7.232296073s) [upload=273839]: Done starting upload Aug 19 16:49:12.801330351 (263h23m7.232410482s) [upload=273839]: Uploading segments... Aug 19 16:49:12.801425372 (263h23m7.232505503s) [upload=273839]: Got next segment. Inline: false Aug 19 16:49:12.801455880 (263h23m7.232536012s) [upload=273839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:49:12.801461982 (263h23m7.232542113s) [upload=273839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:49:13.036708228 (263h23m7.467788359s) [seg_pos={0 0}, upload=273839]: Begin upload segment... Aug 19 16:49:13.036749197 (263h23m7.467829328s) [seg_pos={0 0}, upload=273839]: Done begin upload segment. Aug 19 16:49:13.036771971 (263h23m7.467852092s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.036780256 (263h23m7.467860437s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037135917 (263h23m7.468216048s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037326592 (263h23m7.468406723s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037403259 (263h23m7.46848339s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037667814 (263h23m7.468747945s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037761764 (263h23m7.468841896s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037943331 (263h23m7.469023462s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038070565 (263h23m7.469150696s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038168622 (263h23m7.469248753s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038300104 (263h23m7.469380235s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038396789 (263h23m7.46947692s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038622241 (263h23m7.469702362s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038679350 (263h23m7.469759481s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038854735 (263h23m7.469934856s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.039048045 (263h23m7.470128166s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:14.749064365 (263h23m9.180144496s) [upload=273839]: Next returned nil segment Aug 19 16:49:14.749117217 (263h23m9.180197348s) [upload=273839]: Waiting for error group managing segments... Aug 19 16:49:15.090317863 (263h23m9.521397994s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.132109007 (263h23m9.563189138s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.222090156 (263h23m9.653170287s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.328041780 (263h23m9.759121911s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.353998781 (263h23m9.785078912s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:36.709543777 (263h23m31.140623898s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:37.041062020 (263h23m31.472142141s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.183707791 (263h23m32.614787922s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.694075084 (263h23m33.125155215s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.716788163 (263h23m34.147868294s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.851890717 (263h23m34.282970848s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:40.293306092 (263h23m34.724386243s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:42.156929229 (263h23m36.58800936s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157075889 (263h23m36.58815601s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157126075 (263h23m36.588206206s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection Aug 19 16:49:42.157170951 (263h23m36.588251082s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: read tcp 188.166.225.170:34242->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:42.157215747 (263h23m36.588295878s) [seg_pos={0 0}, upload=273839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 19 16:49:42.385478364 (263h23m36.816558495s) [upload=273839]: Done uploading segments... Aug 19 16:49:42.385503091 (263h23m36.816583212s) [upload=273839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.853231423s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.332247063s FAIL (2025-08-19 17:19:24 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 17:04:06.343826144 (263h38m0.774906265s) [upload=274088]: Starting upload Aug 19 17:04:06.343875278 (263h38m0.774955409s) [upload=274088]: Done starting upload Aug 19 17:04:06.343971703 (263h38m0.775051834s) [upload=274088]: Uploading segments... Aug 19 17:04:06.344028321 (263h38m0.775108453s) [upload=274088]: Got next segment. Inline: true Aug 19 17:04:06.344033120 (263h38m0.775113252s) [upload=274088]: Waiting for error group managing segments... Aug 19 17:04:06.344068187 (263h38m0.775148318s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:04:06.344071964 (263h38m0.775152095s) [upload=274088]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:04:06.344074499 (263h38m0.77515462s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:04:06.653287654 (263h38m1.084367785s) [upload=274088]: Done uploading segments... Aug 19 17:04:06.653300519 (263h38m1.08438065s) [upload=274088]: Upload finished. err: <nil> Duration:* 2.332273734s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X....X.......................................] duration: 5.86966378s FAIL (2025-08-19 17:19:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.86968504s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [............X......X.XX........................X............] duration: 5.909824613s FAIL (2025-08-19 17:19:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== Aug 19 17:09:15.684096389 (263h43m10.11517652s) [upload=274176]: Starting upload Aug 19 17:09:15.684143008 (263h43m10.115223139s) [upload=274176]: Done starting upload Aug 19 17:09:15.684256185 (263h43m10.115336306s) [upload=274176]: Uploading segments... Aug 19 17:09:15.684468862 (263h43m10.115548993s) [upload=274176]: Got next segment. Inline: false Aug 19 17:09:15.684504950 (263h43m10.115585082s) [upload=274176]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:09:15.684509770 (263h43m10.115589901s) [upload=274176]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:09:16.135567491 (263h43m10.566647622s) [seg_pos={0 0}, upload=274176]: Begin upload segment... Aug 19 17:09:16.135613589 (263h43m10.56669372s) [seg_pos={0 0}, upload=274176]: Done begin upload segment. Aug 19 17:09:16.135636573 (263h43m10.566716695s) [noise=true , address=104.36.82.5:4853 , piece_id=K6TRSISY75NXYJNAAEALQ7W4RA26P5TGIVRMHX5SGOIF2GJ7X3GQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.135699574 (263h43m10.566779705s) [noise=true , address=104.36.82.60:4341 , piece_id=5NWVHAFJXPCBGPMRT42H4C3LPE3XKBDKPSX44D3DOWF6XXXJ5LFA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136028283 (263h43m10.567108404s) [noise=true , address=79.127.220.98:4432 , piece_id=4VA6KKCGA5GM55N3SD7FNUILD5XWYHE5VFRBFDIBAGVEK2VHQELA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136206604 (263h43m10.567286735s) [noise=true , address=104.36.82.22:4538 , piece_id=YUOOOPSZLQRNVU557APDO5FTOCY4FEYAHCBZQEE6YWMBPK5ZJYNA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136295745 (263h43m10.567375876s) [noise=true , address=77.90.31.11:4564 , piece_id=TX3C4VLEEOJOHBN7VDK4KIROG42PDDCB3SK26FZA7PM2455YZCMA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136640434 (263h43m10.567720565s) [noise=true , address=104.36.82.16:4315 , piece_id=P3SYICVJ5HSVY4MM62FDXQXJARZLIOQYNPBCT2SKYFPYZ3G3MSWQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136649702 (263h43m10.567729833s) [noise=true , address=104.36.82.44:4874 , piece_id=D3L2BM57MTGKIWKRSSEL7DRXUOK7LLKHL7M3Y3WAFCBILVDEV24Q, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136950888 (263h43m10.568031019s) [noise=true , address=104.36.82.35:4605 , piece_id=ORHH35ESP7KPGGDRD7ANECFHW7YTA2VFZTB5EMRCBQX5EWB7ZU4Q, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136961468 (263h43m10.568041599s) [noise=true , address=77.90.31.10:4535 , piece_id=N6LDVXERZAIR3EZ2TAMFQBM5Y565FMSP44I5KAWC32DTIOOM3FIQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137201027 (263h43m10.568281158s) [noise=true , address=77.90.31.12:4611 , piece_id=BANXKRHIMMOQFBSC4FW2E3SMNXCQERG2OTD7GXMM56T3X6JPNQGQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137260470 (263h43m10.568340601s) [noise=true , address=104.36.82.11:4134 , piece_id=BSGC4ADI6LMK3E2YEYPWFMYS3GC5VA42YM22VIJEGZ2JANYMLBYQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137463418 (263h43m10.568543549s) [noise=true , address=45.91.116.10:4110 , piece_id=KCZZOHPLSRNA45VXOMW7AKUYMRRFRUA52KSOBHNDRJPCB3RMV5HA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137553992 (263h43m10.568634123s) [noise=true , address=104.36.82.55:4224 , piece_id=PGZXUHGOONWDPIB4ZHAKLL4F2UWMZJPAL7BOTXIGSQWESLUPWGQA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137682447 (263h43m10.568762578s) [noise=true , address=45.91.116.12:4238 , piece_id=JAUKDCYIMCIOX4EFHQMCNCBLVZEIBKODPBL5QVBPOJIB554KLNRQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137883032 (263h43m10.568963163s) [noise=true , address=77.90.31.16:4743 , piece_id=YKPMLYPRMAZEBX7ORWEICH5AWKVS2J74D25W4ANIJJOUIBX5UBRA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137912418 (263h43m10.568992539s) [noise=true , address=104.36.82.23:4455 , piece_id=L2M5P5LRKVLVVDSYIOBV4SR3EAH2UO2JUZBGKZKIWBUIGFZPVEOA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.138136216 (263h43m10.569216347s) [noise=true , address=104.36.82.7:4930 , piece_id=EGTR52TZDAVTPPRCJLK4GOUXUWW2U7XJMQOV5P3JCKAKOK7GG5JQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.138374362 (263h43m10.569454493s) [noise=true , address=77.90.31.18:4839 , piece_id=BG553GCZSJEH5ORINSIQ7IUYU3UBQ3WO73ZAGRB7W45HOASJMBVQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:17.616571604 (263h43m12.047651765s) [upload=274176]: Next returned nil segment Aug 19 17:09:17.616591412 (263h43m12.047671543s) [upload=274176]: Waiting for error group managing segments... Aug 19 17:09:18.008239751 (263h43m12.439319882s) [noise=true , address=79.127.220.98:4432 , piece_id=4VA6KKCGA5GM55N3SD7FNUILD5XWYHE5VFRBFDIBAGVEK2VHQELA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.039911371 (263h43m12.470991502s) [noise=true , address=45.91.116.12:4238 , piece_id=JAUKDCYIMCIOX4EFHQMCNCBLVZEIBKODPBL5QVBPOJIB554KLNRQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.065695686 (263h43m12.496775817s) [noise=true , address=77.90.31.16:4743 , piece_id=YKPMLYPRMAZEBX7ORWEICH5AWKVS2J74D25W4ANIJJOUIBX5UBRA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.080085439 (263h43m12.51116557s) [noise=true , address=45.91.116.10:4110 , piece_id=KCZZOHPLSRNA45VXOMW7AKUYMRRFRUA52KSOBHNDRJPCB3RMV5HA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.131454157 (263h43m12.562534288s) [noise=true , address=104.36.82.11:4134 , piece_id=BSGC4ADI6LMK3E2YEYPWFMYS3GC5VA42YM22VIJEGZ2JANYMLBYQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.143557214 (263h43m12.574637345s) [noise=true , address=104.36.82.7:4930 , piece_id=EGTR52TZDAVTPPRCJLK4GOUXUWW2U7XJMQOV5P3JCKAKOK7GG5JQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.156135631 (263h43m12.587215762s) [noise=true , address=77.90.31.12:4611 , piece_id=BANXKRHIMMOQFBSC4FW2E3SMNXCQERG2OTD7GXMM56T3X6JPNQGQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.159345252 (263h43m12.590425373s) [noise=true , address=104.36.82.55:4224 , piece_id=PGZXUHGOONWDPIB4ZHAKLL4F2UWMZJPAL7BOTXIGSQWESLUPWGQA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.169106219 (263h43m12.60018635s) [noise=true , address=104.36.82.22:4538 , piece_id=YUOOOPSZLQRNVU557APDO5FTOCY4FEYAHCBZQEE6YWMBPK5ZJYNA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.188273593 (263h43m12.619353724s) [noise=true , address=104.36.82.35:4605 , piece_id=ORHH35ESP7KPGGDRD7ANECFHW7YTA2VFZTB5EMRCBQX5EWB7ZU4Q, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.204131846 (263h43m12.635211977s) [noise=true , address=104.36.82.16:4315 , piece_id=P3SYICVJ5HSVY4MM62FDXQXJARZLIOQYNPBCT2SKYFPYZ3G3MSWQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.217541965 (263h43m12.648622096s) [noise=true , address=104.36.82.60:4341 , piece_id=5NWVHAFJXPCBGPMRT42H4C3LPE3XKBDKPSX44D3DOWF6XXXJ5LFA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.222565827 (263h43m12.653645948s) [noise=true , address=104.36.82.23:4455 , piece_id=L2M5P5LRKVLVVDSYIOBV4SR3EAH2UO2JUZBGKZKIWBUIGFZPVEOA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.377619011 (263h43m12.808699132s) [noise=true , address=104.36.82.44:4874 , piece_id=D3L2BM57MTGKIWKRSSEL7DRXUOK7LLKHL7M3Y3WAFCBILVDEV24Q, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.377646604 (263h43m12.808726725s) [seg_pos={0 0}, upload=274176]: Segment reached optimal threshold of 14 pieces. Aug 19 17:09:18.377754100 (263h43m12.808834231s) [noise=true , address=104.36.82.5:4853 , piece_id=K6TRSISY75NXYJNAAEALQ7W4RA26P5TGIVRMHX5SGOIF2GJ7X3GQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377895570 (263h43m12.808975701s) [noise=true , address=77.90.31.18:4839 , piece_id=BG553GCZSJEH5ORINSIQ7IUYU3UBQ3WO73ZAGRB7W45HOASJMBVQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gmWwyaQ6zM6QdayBFpXdE9VSE1aaQcBHFcukJoASjjHWCDsNx): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377915529 (263h43m12.80899566s) [noise=true , address=77.90.31.11:4564 , piece_id=TX3C4VLEEOJOHBN7VDK4KIROG42PDDCB3SK26FZA7PM2455YZCMA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377974832 (263h43m12.809054963s) [noise=true , address=77.90.31.10:4535 , piece_id=N6LDVXERZAIR3EZ2TAMFQBM5Y565FMSP44I5KAWC32DTIOOM3FIQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377999008 (263h43m12.80907914s) [seg_pos={0 0}, upload=274176]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 17:09:18.378145599 (263h43m12.80922573s) [upload=274176]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 17:09:18.378150488 (263h43m12.809230619s) [upload=274176]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:09:18.983488776 (263h43m13.414568907s) [upload=274176]: Done uploading segments... Aug 19 17:09:18.983507222 (263h43m13.414587353s) [upload=274176]: Upload finished. err: <nil> Duration: 5.909851325s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 1.994093804s FAIL (2025-08-19 17:18:51 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 17:04:06.343826144 (263h38m0.774906265s) [upload=274088]: Starting upload Aug 19 17:04:06.343875278 (263h38m0.774955409s) [upload=274088]: Done starting upload Aug 19 17:04:06.343971703 (263h38m0.775051834s) [upload=274088]: Uploading segments... Aug 19 17:04:06.344028321 (263h38m0.775108453s) [upload=274088]: Got next segment. Inline: true Aug 19 17:04:06.344033120 (263h38m0.775113252s) [upload=274088]: Waiting for error group managing segments... Aug 19 17:04:06.344068187 (263h38m0.775148318s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:04:06.344071964 (263h38m0.775152095s) [upload=274088]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:04:06.344074499 (263h38m0.77515462s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:04:06.653287654 (263h38m1.084367785s) [upload=274088]: Done uploading segments... Aug 19 17:04:06.653300519 (263h38m1.08438065s) [upload=274088]: Upload finished. err: <nil> Duration:* 1.994130384s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X....XX.......................................] duration: 6.094048015s FAIL (2025-08-19 17:18:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 19 16:49:12.801154313 (263h23m7.232234445s) [upload=273839]: Starting upload Aug 19 16:49:12.801215942 (263h23m7.232296073s) [upload=273839]: Done starting upload Aug 19 16:49:12.801330351 (263h23m7.232410482s) [upload=273839]: Uploading segments... Aug 19 16:49:12.801425372 (263h23m7.232505503s) [upload=273839]: Got next segment. Inline: false Aug 19 16:49:12.801455880 (263h23m7.232536012s) [upload=273839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:49:12.801461982 (263h23m7.232542113s) [upload=273839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:49:13.036708228 (263h23m7.467788359s) [seg_pos={0 0}, upload=273839]: Begin upload segment... Aug 19 16:49:13.036749197 (263h23m7.467829328s) [seg_pos={0 0}, upload=273839]: Done begin upload segment. Aug 19 16:49:13.036771971 (263h23m7.467852092s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.036780256 (263h23m7.467860437s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037135917 (263h23m7.468216048s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037326592 (263h23m7.468406723s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037403259 (263h23m7.46848339s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037667814 (263h23m7.468747945s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037761764 (263h23m7.468841896s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037943331 (263h23m7.469023462s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038070565 (263h23m7.469150696s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038168622 (263h23m7.469248753s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038300104 (263h23m7.469380235s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038396789 (263h23m7.46947692s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038622241 (263h23m7.469702362s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038679350 (263h23m7.469759481s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038854735 (263h23m7.469934856s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.039048045 (263h23m7.470128166s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:14.749064365 (263h23m9.180144496s) [upload=273839]: Next returned nil segment Aug 19 16:49:14.749117217 (263h23m9.180197348s) [upload=273839]: Waiting for error group managing segments... Aug 19 16:49:15.090317863 (263h23m9.521397994s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.132109007 (263h23m9.563189138s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.222090156 (263h23m9.653170287s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.328041780 (263h23m9.759121911s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.353998781 (263h23m9.785078912s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:36.709543777 (263h23m31.140623898s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:37.041062020 (263h23m31.472142141s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.183707791 (263h23m32.614787922s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.694075084 (263h23m33.125155215s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.716788163 (263h23m34.147868294s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.851890717 (263h23m34.282970848s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:40.293306092 (263h23m34.724386243s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:42.156929229 (263h23m36.58800936s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157075889 (263h23m36.58815601s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157126075 (263h23m36.588206206s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection Aug 19 16:49:42.157170951 (263h23m36.588251082s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: read tcp 188.166.225.170:34242->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:42.157215747 (263h23m36.588295878s) [seg_pos={0 0}, upload=273839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 19 16:49:42.385478364 (263h23m36.816558495s) [upload=273839]: Done uploading segments... Aug 19 16:49:42.385503091 (263h23m36.816583212s) [upload=273839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.094076599s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.649908283s FAIL (2025-08-19 17:18:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 263h18m33.75518017s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324344006 (263h18m33.755424127s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324633220 (263h18m33.755713351s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324851286 (263h18m33.755931417s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325068942 (263h18m33.756149063s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325283613 (263h18m33.756363744s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325499857 (263h18m33.756579998s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325759794 (263h18m33.756839915s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326059428 (263h18m33.757139549s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326277565 (263h18m33.757357696s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326470835 (263h18m33.757550966s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326673753 (263h18m33.757753884s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326867664 (263h18m33.757947795s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:40.851885389 (263h18m35.28296552s) [upload=273764]: Next returned nil segment Aug 19 16:44:40.851903334 (263h18m35.282983465s) [upload=273764]: Waiting for error group managing segments... Aug 19 16:44:41.244668372 (263h18m35.675748503s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.277280365 (263h18m35.708360496s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.328561665 (263h18m35.759641786s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.331747139 (263h18m35.76282727s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.357720320 (263h18m35.788800481s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.378190805 (263h18m35.809270936s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.486480333 (263h18m35.917560464s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:45:38.230570787 (263h19m32.661650918s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: expected piece hash; context deadline exceeded; EOF Aug 19 16:45:38.230979869 (263h19m32.66206s) [noise=true , address=77.90.31.18:4840 , piece_id=JWOETSFSZ3HELLZNLEASYNGW5OR64TMQVOFYYJAJVTHWCBHVTSLQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: read tcp 188.166.225.170:39946->77.90.31.18:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231017812 (263h19m32.662097943s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 188.166.225.170:44164->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231063489 (263h19m32.66214362s) [noise=true , address=45.91.116.14:4203 , piece_id=WRLBES3FI5EA5AQLZXJYGDROP3KBG3E2F22R4YD6A5KKI44PLV2Q, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 188.166.225.170:51738->45.91.116.14:4203: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231090952 (263h19m32.662171083s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: read tcp 188.166.225.170:57158->77.90.31.15:4707: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231110719 (263h19m32.66219085s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:50784->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231140165 (263h19m32.662220296s) [noise=true , address=77.90.31.16:4749 , piece_id=XCPXVBYLQH345FPIW4KDNLBFUCDDEW2H4B25WIXGOO5FVMAIVKPQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: read tcp 188.166.225.170:51668->77.90.31.16:4749: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231144814 (263h19m32.662224945s) [noise=true , address=77.90.31.19:4865 , piece_id=B3HUFIJZKVZJH4OBBO2DD7S3I7HX4LO2IQ74XJPO4DTW4QER35NA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1bguBghQonV5sTHaWteyUEnpp5KNzBKbvBpbvrUNFDCveHN77X, address:77.90.31.19:4865): protocol: read tcp 188.166.225.170:39398->77.90.31.19:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231177167 (263h19m32.662257298s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: read tcp 188.166.225.170:58526->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231184571 (263h19m32.662264702s) [noise=true , address=45.91.116.12:4236 , piece_id=LCRQS2GOSHOSLZBGQTGB3Q3PA4VN7KXNW66TDZ5O424TPJWTQE3A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH, address:45.91.116.12:4236): protocol: read tcp 188.166.225.170:57166->45.91.116.12:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231189770 (263h19m32.662269901s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 188.166.225.170:54242->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231215850 (263h19m32.662295981s) [seg_pos={0 0}, upload=273764]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 19 16:45:38.459100173 (263h19m32.890180304s) [upload=273764]: Done uploading segments... Aug 19 16:45:38.459128316 (263h19m32.890208447s) [upload=273764]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.649935126s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X....X......................................] duration: 6.078238685s FAIL (2025-08-19 17:18:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.078253242s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............X...X........................................] duration: 6.638538106s FAIL (2025-08-19 17:18:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.63856051s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 1.971086031s FAIL (2025-08-19 17:18:26 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 17:04:06.343826144 (263h38m0.774906265s) [upload=274088]: Starting upload Aug 19 17:04:06.343875278 (263h38m0.774955409s) [upload=274088]: Done starting upload Aug 19 17:04:06.343971703 (263h38m0.775051834s) [upload=274088]: Uploading segments... Aug 19 17:04:06.344028321 (263h38m0.775108453s) [upload=274088]: Got next segment. Inline: true Aug 19 17:04:06.344033120 (263h38m0.775113252s) [upload=274088]: Waiting for error group managing segments... Aug 19 17:04:06.344068187 (263h38m0.775148318s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:04:06.344071964 (263h38m0.775152095s) [upload=274088]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:04:06.344074499 (263h38m0.77515462s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:04:06.653287654 (263h38m1.084367785s) [upload=274088]: Done uploading segments... Aug 19 17:04:06.653300519 (263h38m1.08438065s) [upload=274088]: Upload finished. err: <nil> Duration:* 1.971112572s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............X......X.XX........................X...........] duration: 5.929235357s FAIL (2025-08-19 17:18:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== Aug 19 17:09:15.684096389 (263h43m10.11517652s) [upload=274176]: Starting upload Aug 19 17:09:15.684143008 (263h43m10.115223139s) [upload=274176]: Done starting upload Aug 19 17:09:15.684256185 (263h43m10.115336306s) [upload=274176]: Uploading segments... Aug 19 17:09:15.684468862 (263h43m10.115548993s) [upload=274176]: Got next segment. Inline: false Aug 19 17:09:15.684504950 (263h43m10.115585082s) [upload=274176]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:09:15.684509770 (263h43m10.115589901s) [upload=274176]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:09:16.135567491 (263h43m10.566647622s) [seg_pos={0 0}, upload=274176]: Begin upload segment... Aug 19 17:09:16.135613589 (263h43m10.56669372s) [seg_pos={0 0}, upload=274176]: Done begin upload segment. Aug 19 17:09:16.135636573 (263h43m10.566716695s) [noise=true , address=104.36.82.5:4853 , piece_id=K6TRSISY75NXYJNAAEALQ7W4RA26P5TGIVRMHX5SGOIF2GJ7X3GQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.135699574 (263h43m10.566779705s) [noise=true , address=104.36.82.60:4341 , piece_id=5NWVHAFJXPCBGPMRT42H4C3LPE3XKBDKPSX44D3DOWF6XXXJ5LFA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136028283 (263h43m10.567108404s) [noise=true , address=79.127.220.98:4432 , piece_id=4VA6KKCGA5GM55N3SD7FNUILD5XWYHE5VFRBFDIBAGVEK2VHQELA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136206604 (263h43m10.567286735s) [noise=true , address=104.36.82.22:4538 , piece_id=YUOOOPSZLQRNVU557APDO5FTOCY4FEYAHCBZQEE6YWMBPK5ZJYNA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136295745 (263h43m10.567375876s) [noise=true , address=77.90.31.11:4564 , piece_id=TX3C4VLEEOJOHBN7VDK4KIROG42PDDCB3SK26FZA7PM2455YZCMA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136640434 (263h43m10.567720565s) [noise=true , address=104.36.82.16:4315 , piece_id=P3SYICVJ5HSVY4MM62FDXQXJARZLIOQYNPBCT2SKYFPYZ3G3MSWQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136649702 (263h43m10.567729833s) [noise=true , address=104.36.82.44:4874 , piece_id=D3L2BM57MTGKIWKRSSEL7DRXUOK7LLKHL7M3Y3WAFCBILVDEV24Q, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136950888 (263h43m10.568031019s) [noise=true , address=104.36.82.35:4605 , piece_id=ORHH35ESP7KPGGDRD7ANECFHW7YTA2VFZTB5EMRCBQX5EWB7ZU4Q, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136961468 (263h43m10.568041599s) [noise=true , address=77.90.31.10:4535 , piece_id=N6LDVXERZAIR3EZ2TAMFQBM5Y565FMSP44I5KAWC32DTIOOM3FIQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137201027 (263h43m10.568281158s) [noise=true , address=77.90.31.12:4611 , piece_id=BANXKRHIMMOQFBSC4FW2E3SMNXCQERG2OTD7GXMM56T3X6JPNQGQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137260470 (263h43m10.568340601s) [noise=true , address=104.36.82.11:4134 , piece_id=BSGC4ADI6LMK3E2YEYPWFMYS3GC5VA42YM22VIJEGZ2JANYMLBYQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137463418 (263h43m10.568543549s) [noise=true , address=45.91.116.10:4110 , piece_id=KCZZOHPLSRNA45VXOMW7AKUYMRRFRUA52KSOBHNDRJPCB3RMV5HA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137553992 (263h43m10.568634123s) [noise=true , address=104.36.82.55:4224 , piece_id=PGZXUHGOONWDPIB4ZHAKLL4F2UWMZJPAL7BOTXIGSQWESLUPWGQA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137682447 (263h43m10.568762578s) [noise=true , address=45.91.116.12:4238 , piece_id=JAUKDCYIMCIOX4EFHQMCNCBLVZEIBKODPBL5QVBPOJIB554KLNRQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137883032 (263h43m10.568963163s) [noise=true , address=77.90.31.16:4743 , piece_id=YKPMLYPRMAZEBX7ORWEICH5AWKVS2J74D25W4ANIJJOUIBX5UBRA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137912418 (263h43m10.568992539s) [noise=true , address=104.36.82.23:4455 , piece_id=L2M5P5LRKVLVVDSYIOBV4SR3EAH2UO2JUZBGKZKIWBUIGFZPVEOA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.138136216 (263h43m10.569216347s) [noise=true , address=104.36.82.7:4930 , piece_id=EGTR52TZDAVTPPRCJLK4GOUXUWW2U7XJMQOV5P3JCKAKOK7GG5JQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.138374362 (263h43m10.569454493s) [noise=true , address=77.90.31.18:4839 , piece_id=BG553GCZSJEH5ORINSIQ7IUYU3UBQ3WO73ZAGRB7W45HOASJMBVQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:17.616571604 (263h43m12.047651765s) [upload=274176]: Next returned nil segment Aug 19 17:09:17.616591412 (263h43m12.047671543s) [upload=274176]: Waiting for error group managing segments... Aug 19 17:09:18.008239751 (263h43m12.439319882s) [noise=true , address=79.127.220.98:4432 , piece_id=4VA6KKCGA5GM55N3SD7FNUILD5XWYHE5VFRBFDIBAGVEK2VHQELA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.039911371 (263h43m12.470991502s) [noise=true , address=45.91.116.12:4238 , piece_id=JAUKDCYIMCIOX4EFHQMCNCBLVZEIBKODPBL5QVBPOJIB554KLNRQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.065695686 (263h43m12.496775817s) [noise=true , address=77.90.31.16:4743 , piece_id=YKPMLYPRMAZEBX7ORWEICH5AWKVS2J74D25W4ANIJJOUIBX5UBRA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.080085439 (263h43m12.51116557s) [noise=true , address=45.91.116.10:4110 , piece_id=KCZZOHPLSRNA45VXOMW7AKUYMRRFRUA52KSOBHNDRJPCB3RMV5HA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.131454157 (263h43m12.562534288s) [noise=true , address=104.36.82.11:4134 , piece_id=BSGC4ADI6LMK3E2YEYPWFMYS3GC5VA42YM22VIJEGZ2JANYMLBYQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.143557214 (263h43m12.574637345s) [noise=true , address=104.36.82.7:4930 , piece_id=EGTR52TZDAVTPPRCJLK4GOUXUWW2U7XJMQOV5P3JCKAKOK7GG5JQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.156135631 (263h43m12.587215762s) [noise=true , address=77.90.31.12:4611 , piece_id=BANXKRHIMMOQFBSC4FW2E3SMNXCQERG2OTD7GXMM56T3X6JPNQGQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.159345252 (263h43m12.590425373s) [noise=true , address=104.36.82.55:4224 , piece_id=PGZXUHGOONWDPIB4ZHAKLL4F2UWMZJPAL7BOTXIGSQWESLUPWGQA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.169106219 (263h43m12.60018635s) [noise=true , address=104.36.82.22:4538 , piece_id=YUOOOPSZLQRNVU557APDO5FTOCY4FEYAHCBZQEE6YWMBPK5ZJYNA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.188273593 (263h43m12.619353724s) [noise=true , address=104.36.82.35:4605 , piece_id=ORHH35ESP7KPGGDRD7ANECFHW7YTA2VFZTB5EMRCBQX5EWB7ZU4Q, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.204131846 (263h43m12.635211977s) [noise=true , address=104.36.82.16:4315 , piece_id=P3SYICVJ5HSVY4MM62FDXQXJARZLIOQYNPBCT2SKYFPYZ3G3MSWQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.217541965 (263h43m12.648622096s) [noise=true , address=104.36.82.60:4341 , piece_id=5NWVHAFJXPCBGPMRT42H4C3LPE3XKBDKPSX44D3DOWF6XXXJ5LFA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.222565827 (263h43m12.653645948s) [noise=true , address=104.36.82.23:4455 , piece_id=L2M5P5LRKVLVVDSYIOBV4SR3EAH2UO2JUZBGKZKIWBUIGFZPVEOA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.377619011 (263h43m12.808699132s) [noise=true , address=104.36.82.44:4874 , piece_id=D3L2BM57MTGKIWKRSSEL7DRXUOK7LLKHL7M3Y3WAFCBILVDEV24Q, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.377646604 (263h43m12.808726725s) [seg_pos={0 0}, upload=274176]: Segment reached optimal threshold of 14 pieces. Aug 19 17:09:18.377754100 (263h43m12.808834231s) [noise=true , address=104.36.82.5:4853 , piece_id=K6TRSISY75NXYJNAAEALQ7W4RA26P5TGIVRMHX5SGOIF2GJ7X3GQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377895570 (263h43m12.808975701s) [noise=true , address=77.90.31.18:4839 , piece_id=BG553GCZSJEH5ORINSIQ7IUYU3UBQ3WO73ZAGRB7W45HOASJMBVQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gmWwyaQ6zM6QdayBFpXdE9VSE1aaQcBHFcukJoASjjHWCDsNx): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377915529 (263h43m12.80899566s) [noise=true , address=77.90.31.11:4564 , piece_id=TX3C4VLEEOJOHBN7VDK4KIROG42PDDCB3SK26FZA7PM2455YZCMA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377974832 (263h43m12.809054963s) [noise=true , address=77.90.31.10:4535 , piece_id=N6LDVXERZAIR3EZ2TAMFQBM5Y565FMSP44I5KAWC32DTIOOM3FIQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377999008 (263h43m12.80907914s) [seg_pos={0 0}, upload=274176]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 17:09:18.378145599 (263h43m12.80922573s) [upload=274176]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 17:09:18.378150488 (263h43m12.809230619s) [upload=274176]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:09:18.983488776 (263h43m13.414568907s) [upload=274176]: Done uploading segments... Aug 19 17:09:18.983507222 (263h43m13.414587353s) [upload=274176]: Upload finished. err: <nil> Duration: 5.929271044s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X....XX......................................] duration: 5.802366059s FAIL (2025-08-19 17:17:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 19 16:49:12.801154313 (263h23m7.232234445s) [upload=273839]: Starting upload Aug 19 16:49:12.801215942 (263h23m7.232296073s) [upload=273839]: Done starting upload Aug 19 16:49:12.801330351 (263h23m7.232410482s) [upload=273839]: Uploading segments... Aug 19 16:49:12.801425372 (263h23m7.232505503s) [upload=273839]: Got next segment. Inline: false Aug 19 16:49:12.801455880 (263h23m7.232536012s) [upload=273839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:49:12.801461982 (263h23m7.232542113s) [upload=273839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:49:13.036708228 (263h23m7.467788359s) [seg_pos={0 0}, upload=273839]: Begin upload segment... Aug 19 16:49:13.036749197 (263h23m7.467829328s) [seg_pos={0 0}, upload=273839]: Done begin upload segment. Aug 19 16:49:13.036771971 (263h23m7.467852092s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.036780256 (263h23m7.467860437s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037135917 (263h23m7.468216048s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037326592 (263h23m7.468406723s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037403259 (263h23m7.46848339s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037667814 (263h23m7.468747945s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037761764 (263h23m7.468841896s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037943331 (263h23m7.469023462s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038070565 (263h23m7.469150696s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038168622 (263h23m7.469248753s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038300104 (263h23m7.469380235s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038396789 (263h23m7.46947692s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038622241 (263h23m7.469702362s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038679350 (263h23m7.469759481s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038854735 (263h23m7.469934856s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.039048045 (263h23m7.470128166s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:14.749064365 (263h23m9.180144496s) [upload=273839]: Next returned nil segment Aug 19 16:49:14.749117217 (263h23m9.180197348s) [upload=273839]: Waiting for error group managing segments... Aug 19 16:49:15.090317863 (263h23m9.521397994s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.132109007 (263h23m9.563189138s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.222090156 (263h23m9.653170287s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.328041780 (263h23m9.759121911s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.353998781 (263h23m9.785078912s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:36.709543777 (263h23m31.140623898s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:37.041062020 (263h23m31.472142141s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.183707791 (263h23m32.614787922s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.694075084 (263h23m33.125155215s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.716788163 (263h23m34.147868294s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.851890717 (263h23m34.282970848s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:40.293306092 (263h23m34.724386243s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:42.156929229 (263h23m36.58800936s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157075889 (263h23m36.58815601s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157126075 (263h23m36.588206206s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection Aug 19 16:49:42.157170951 (263h23m36.588251082s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: read tcp 188.166.225.170:34242->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:42.157215747 (263h23m36.588295878s) [seg_pos={0 0}, upload=273839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 19 16:49:42.385478364 (263h23m36.816558495s) [upload=273839]: Done uploading segments... Aug 19 16:49:42.385503091 (263h23m36.816583212s) [upload=273839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.80238751s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X....X.....................................] duration: 5.928544657s FAIL (2025-08-19 17:17:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.92856278s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 1.914006268s FAIL (2025-08-19 17:17:38 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 17:04:06.343826144 (263h38m0.774906265s) [upload=274088]: Starting upload Aug 19 17:04:06.343875278 (263h38m0.774955409s) [upload=274088]: Done starting upload Aug 19 17:04:06.343971703 (263h38m0.775051834s) [upload=274088]: Uploading segments... Aug 19 17:04:06.344028321 (263h38m0.775108453s) [upload=274088]: Got next segment. Inline: true Aug 19 17:04:06.344033120 (263h38m0.775113252s) [upload=274088]: Waiting for error group managing segments... Aug 19 17:04:06.344068187 (263h38m0.775148318s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:04:06.344071964 (263h38m0.775152095s) [upload=274088]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:04:06.344074499 (263h38m0.77515462s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:04:06.653287654 (263h38m1.084367785s) [upload=274088]: Done uploading segments... Aug 19 17:04:06.653300519 (263h38m1.08438065s) [upload=274088]: Upload finished. err: <nil> Duration:* 1.914052978s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.635201303s FAIL (2025-08-19 17:17:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 263h18m33.75518017s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324344006 (263h18m33.755424127s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324633220 (263h18m33.755713351s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324851286 (263h18m33.755931417s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325068942 (263h18m33.756149063s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325283613 (263h18m33.756363744s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325499857 (263h18m33.756579998s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325759794 (263h18m33.756839915s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326059428 (263h18m33.757139549s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326277565 (263h18m33.757357696s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326470835 (263h18m33.757550966s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326673753 (263h18m33.757753884s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326867664 (263h18m33.757947795s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:40.851885389 (263h18m35.28296552s) [upload=273764]: Next returned nil segment Aug 19 16:44:40.851903334 (263h18m35.282983465s) [upload=273764]: Waiting for error group managing segments... Aug 19 16:44:41.244668372 (263h18m35.675748503s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.277280365 (263h18m35.708360496s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.328561665 (263h18m35.759641786s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.331747139 (263h18m35.76282727s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.357720320 (263h18m35.788800481s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.378190805 (263h18m35.809270936s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.486480333 (263h18m35.917560464s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:45:38.230570787 (263h19m32.661650918s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: expected piece hash; context deadline exceeded; EOF Aug 19 16:45:38.230979869 (263h19m32.66206s) [noise=true , address=77.90.31.18:4840 , piece_id=JWOETSFSZ3HELLZNLEASYNGW5OR64TMQVOFYYJAJVTHWCBHVTSLQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: read tcp 188.166.225.170:39946->77.90.31.18:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231017812 (263h19m32.662097943s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 188.166.225.170:44164->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231063489 (263h19m32.66214362s) [noise=true , address=45.91.116.14:4203 , piece_id=WRLBES3FI5EA5AQLZXJYGDROP3KBG3E2F22R4YD6A5KKI44PLV2Q, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 188.166.225.170:51738->45.91.116.14:4203: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231090952 (263h19m32.662171083s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: read tcp 188.166.225.170:57158->77.90.31.15:4707: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231110719 (263h19m32.66219085s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:50784->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231140165 (263h19m32.662220296s) [noise=true , address=77.90.31.16:4749 , piece_id=XCPXVBYLQH345FPIW4KDNLBFUCDDEW2H4B25WIXGOO5FVMAIVKPQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: read tcp 188.166.225.170:51668->77.90.31.16:4749: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231144814 (263h19m32.662224945s) [noise=true , address=77.90.31.19:4865 , piece_id=B3HUFIJZKVZJH4OBBO2DD7S3I7HX4LO2IQ74XJPO4DTW4QER35NA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1bguBghQonV5sTHaWteyUEnpp5KNzBKbvBpbvrUNFDCveHN77X, address:77.90.31.19:4865): protocol: read tcp 188.166.225.170:39398->77.90.31.19:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231177167 (263h19m32.662257298s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: read tcp 188.166.225.170:58526->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231184571 (263h19m32.662264702s) [noise=true , address=45.91.116.12:4236 , piece_id=LCRQS2GOSHOSLZBGQTGB3Q3PA4VN7KXNW66TDZ5O424TPJWTQE3A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH, address:45.91.116.12:4236): protocol: read tcp 188.166.225.170:57166->45.91.116.12:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231189770 (263h19m32.662269901s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 188.166.225.170:54242->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231215850 (263h19m32.662295981s) [seg_pos={0 0}, upload=273764]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 19 16:45:38.459100173 (263h19m32.890180304s) [upload=273764]: Done uploading segments... Aug 19 16:45:38.459128316 (263h19m32.890208447s) [upload=273764]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.635228414s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X....X....................................] duration: 5.910436773s FAIL (2025-08-19 17:17:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.910457001s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................X...X.......................................] duration: 6.325550302s FAIL (2025-08-19 17:17:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.325580309s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............X......X.XX........................X..........] duration: 5.923363801s FAIL (2025-08-19 17:17:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== Aug 19 17:09:15.684096389 (263h43m10.11517652s) [upload=274176]: Starting upload Aug 19 17:09:15.684143008 (263h43m10.115223139s) [upload=274176]: Done starting upload Aug 19 17:09:15.684256185 (263h43m10.115336306s) [upload=274176]: Uploading segments... Aug 19 17:09:15.684468862 (263h43m10.115548993s) [upload=274176]: Got next segment. Inline: false Aug 19 17:09:15.684504950 (263h43m10.115585082s) [upload=274176]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:09:15.684509770 (263h43m10.115589901s) [upload=274176]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:09:16.135567491 (263h43m10.566647622s) [seg_pos={0 0}, upload=274176]: Begin upload segment... Aug 19 17:09:16.135613589 (263h43m10.56669372s) [seg_pos={0 0}, upload=274176]: Done begin upload segment. Aug 19 17:09:16.135636573 (263h43m10.566716695s) [noise=true , address=104.36.82.5:4853 , piece_id=K6TRSISY75NXYJNAAEALQ7W4RA26P5TGIVRMHX5SGOIF2GJ7X3GQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.135699574 (263h43m10.566779705s) [noise=true , address=104.36.82.60:4341 , piece_id=5NWVHAFJXPCBGPMRT42H4C3LPE3XKBDKPSX44D3DOWF6XXXJ5LFA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136028283 (263h43m10.567108404s) [noise=true , address=79.127.220.98:4432 , piece_id=4VA6KKCGA5GM55N3SD7FNUILD5XWYHE5VFRBFDIBAGVEK2VHQELA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136206604 (263h43m10.567286735s) [noise=true , address=104.36.82.22:4538 , piece_id=YUOOOPSZLQRNVU557APDO5FTOCY4FEYAHCBZQEE6YWMBPK5ZJYNA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136295745 (263h43m10.567375876s) [noise=true , address=77.90.31.11:4564 , piece_id=TX3C4VLEEOJOHBN7VDK4KIROG42PDDCB3SK26FZA7PM2455YZCMA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136640434 (263h43m10.567720565s) [noise=true , address=104.36.82.16:4315 , piece_id=P3SYICVJ5HSVY4MM62FDXQXJARZLIOQYNPBCT2SKYFPYZ3G3MSWQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136649702 (263h43m10.567729833s) [noise=true , address=104.36.82.44:4874 , piece_id=D3L2BM57MTGKIWKRSSEL7DRXUOK7LLKHL7M3Y3WAFCBILVDEV24Q, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136950888 (263h43m10.568031019s) [noise=true , address=104.36.82.35:4605 , piece_id=ORHH35ESP7KPGGDRD7ANECFHW7YTA2VFZTB5EMRCBQX5EWB7ZU4Q, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136961468 (263h43m10.568041599s) [noise=true , address=77.90.31.10:4535 , piece_id=N6LDVXERZAIR3EZ2TAMFQBM5Y565FMSP44I5KAWC32DTIOOM3FIQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137201027 (263h43m10.568281158s) [noise=true , address=77.90.31.12:4611 , piece_id=BANXKRHIMMOQFBSC4FW2E3SMNXCQERG2OTD7GXMM56T3X6JPNQGQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137260470 (263h43m10.568340601s) [noise=true , address=104.36.82.11:4134 , piece_id=BSGC4ADI6LMK3E2YEYPWFMYS3GC5VA42YM22VIJEGZ2JANYMLBYQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137463418 (263h43m10.568543549s) [noise=true , address=45.91.116.10:4110 , piece_id=KCZZOHPLSRNA45VXOMW7AKUYMRRFRUA52KSOBHNDRJPCB3RMV5HA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137553992 (263h43m10.568634123s) [noise=true , address=104.36.82.55:4224 , piece_id=PGZXUHGOONWDPIB4ZHAKLL4F2UWMZJPAL7BOTXIGSQWESLUPWGQA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137682447 (263h43m10.568762578s) [noise=true , address=45.91.116.12:4238 , piece_id=JAUKDCYIMCIOX4EFHQMCNCBLVZEIBKODPBL5QVBPOJIB554KLNRQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137883032 (263h43m10.568963163s) [noise=true , address=77.90.31.16:4743 , piece_id=YKPMLYPRMAZEBX7ORWEICH5AWKVS2J74D25W4ANIJJOUIBX5UBRA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137912418 (263h43m10.568992539s) [noise=true , address=104.36.82.23:4455 , piece_id=L2M5P5LRKVLVVDSYIOBV4SR3EAH2UO2JUZBGKZKIWBUIGFZPVEOA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.138136216 (263h43m10.569216347s) [noise=true , address=104.36.82.7:4930 , piece_id=EGTR52TZDAVTPPRCJLK4GOUXUWW2U7XJMQOV5P3JCKAKOK7GG5JQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.138374362 (263h43m10.569454493s) [noise=true , address=77.90.31.18:4839 , piece_id=BG553GCZSJEH5ORINSIQ7IUYU3UBQ3WO73ZAGRB7W45HOASJMBVQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:17.616571604 (263h43m12.047651765s) [upload=274176]: Next returned nil segment Aug 19 17:09:17.616591412 (263h43m12.047671543s) [upload=274176]: Waiting for error group managing segments... Aug 19 17:09:18.008239751 (263h43m12.439319882s) [noise=true , address=79.127.220.98:4432 , piece_id=4VA6KKCGA5GM55N3SD7FNUILD5XWYHE5VFRBFDIBAGVEK2VHQELA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.039911371 (263h43m12.470991502s) [noise=true , address=45.91.116.12:4238 , piece_id=JAUKDCYIMCIOX4EFHQMCNCBLVZEIBKODPBL5QVBPOJIB554KLNRQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.065695686 (263h43m12.496775817s) [noise=true , address=77.90.31.16:4743 , piece_id=YKPMLYPRMAZEBX7ORWEICH5AWKVS2J74D25W4ANIJJOUIBX5UBRA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.080085439 (263h43m12.51116557s) [noise=true , address=45.91.116.10:4110 , piece_id=KCZZOHPLSRNA45VXOMW7AKUYMRRFRUA52KSOBHNDRJPCB3RMV5HA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.131454157 (263h43m12.562534288s) [noise=true , address=104.36.82.11:4134 , piece_id=BSGC4ADI6LMK3E2YEYPWFMYS3GC5VA42YM22VIJEGZ2JANYMLBYQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.143557214 (263h43m12.574637345s) [noise=true , address=104.36.82.7:4930 , piece_id=EGTR52TZDAVTPPRCJLK4GOUXUWW2U7XJMQOV5P3JCKAKOK7GG5JQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.156135631 (263h43m12.587215762s) [noise=true , address=77.90.31.12:4611 , piece_id=BANXKRHIMMOQFBSC4FW2E3SMNXCQERG2OTD7GXMM56T3X6JPNQGQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.159345252 (263h43m12.590425373s) [noise=true , address=104.36.82.55:4224 , piece_id=PGZXUHGOONWDPIB4ZHAKLL4F2UWMZJPAL7BOTXIGSQWESLUPWGQA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.169106219 (263h43m12.60018635s) [noise=true , address=104.36.82.22:4538 , piece_id=YUOOOPSZLQRNVU557APDO5FTOCY4FEYAHCBZQEE6YWMBPK5ZJYNA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.188273593 (263h43m12.619353724s) [noise=true , address=104.36.82.35:4605 , piece_id=ORHH35ESP7KPGGDRD7ANECFHW7YTA2VFZTB5EMRCBQX5EWB7ZU4Q, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.204131846 (263h43m12.635211977s) [noise=true , address=104.36.82.16:4315 , piece_id=P3SYICVJ5HSVY4MM62FDXQXJARZLIOQYNPBCT2SKYFPYZ3G3MSWQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.217541965 (263h43m12.648622096s) [noise=true , address=104.36.82.60:4341 , piece_id=5NWVHAFJXPCBGPMRT42H4C3LPE3XKBDKPSX44D3DOWF6XXXJ5LFA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.222565827 (263h43m12.653645948s) [noise=true , address=104.36.82.23:4455 , piece_id=L2M5P5LRKVLVVDSYIOBV4SR3EAH2UO2JUZBGKZKIWBUIGFZPVEOA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.377619011 (263h43m12.808699132s) [noise=true , address=104.36.82.44:4874 , piece_id=D3L2BM57MTGKIWKRSSEL7DRXUOK7LLKHL7M3Y3WAFCBILVDEV24Q, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.377646604 (263h43m12.808726725s) [seg_pos={0 0}, upload=274176]: Segment reached optimal threshold of 14 pieces. Aug 19 17:09:18.377754100 (263h43m12.808834231s) [noise=true , address=104.36.82.5:4853 , piece_id=K6TRSISY75NXYJNAAEALQ7W4RA26P5TGIVRMHX5SGOIF2GJ7X3GQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377895570 (263h43m12.808975701s) [noise=true , address=77.90.31.18:4839 , piece_id=BG553GCZSJEH5ORINSIQ7IUYU3UBQ3WO73ZAGRB7W45HOASJMBVQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gmWwyaQ6zM6QdayBFpXdE9VSE1aaQcBHFcukJoASjjHWCDsNx): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377915529 (263h43m12.80899566s) [noise=true , address=77.90.31.11:4564 , piece_id=TX3C4VLEEOJOHBN7VDK4KIROG42PDDCB3SK26FZA7PM2455YZCMA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377974832 (263h43m12.809054963s) [noise=true , address=77.90.31.10:4535 , piece_id=N6LDVXERZAIR3EZ2TAMFQBM5Y565FMSP44I5KAWC32DTIOOM3FIQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377999008 (263h43m12.80907914s) [seg_pos={0 0}, upload=274176]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 17:09:18.378145599 (263h43m12.80922573s) [upload=274176]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 17:09:18.378150488 (263h43m12.809230619s) [upload=274176]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:09:18.983488776 (263h43m13.414568907s) [upload=274176]: Done uploading segments... Aug 19 17:09:18.983507222 (263h43m13.414587353s) [upload=274176]: Upload finished. err: <nil> Duration: 5.923395763s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X....XX.....................................] duration: 6.041448936s FAIL (2025-08-19 17:16:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 19 16:49:12.801154313 (263h23m7.232234445s) [upload=273839]: Starting upload Aug 19 16:49:12.801215942 (263h23m7.232296073s) [upload=273839]: Done starting upload Aug 19 16:49:12.801330351 (263h23m7.232410482s) [upload=273839]: Uploading segments... Aug 19 16:49:12.801425372 (263h23m7.232505503s) [upload=273839]: Got next segment. Inline: false Aug 19 16:49:12.801455880 (263h23m7.232536012s) [upload=273839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:49:12.801461982 (263h23m7.232542113s) [upload=273839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:49:13.036708228 (263h23m7.467788359s) [seg_pos={0 0}, upload=273839]: Begin upload segment... Aug 19 16:49:13.036749197 (263h23m7.467829328s) [seg_pos={0 0}, upload=273839]: Done begin upload segment. Aug 19 16:49:13.036771971 (263h23m7.467852092s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.036780256 (263h23m7.467860437s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037135917 (263h23m7.468216048s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037326592 (263h23m7.468406723s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037403259 (263h23m7.46848339s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037667814 (263h23m7.468747945s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037761764 (263h23m7.468841896s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037943331 (263h23m7.469023462s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038070565 (263h23m7.469150696s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038168622 (263h23m7.469248753s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038300104 (263h23m7.469380235s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038396789 (263h23m7.46947692s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038622241 (263h23m7.469702362s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038679350 (263h23m7.469759481s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038854735 (263h23m7.469934856s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.039048045 (263h23m7.470128166s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:14.749064365 (263h23m9.180144496s) [upload=273839]: Next returned nil segment Aug 19 16:49:14.749117217 (263h23m9.180197348s) [upload=273839]: Waiting for error group managing segments... Aug 19 16:49:15.090317863 (263h23m9.521397994s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.132109007 (263h23m9.563189138s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.222090156 (263h23m9.653170287s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.328041780 (263h23m9.759121911s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.353998781 (263h23m9.785078912s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:36.709543777 (263h23m31.140623898s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:37.041062020 (263h23m31.472142141s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.183707791 (263h23m32.614787922s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.694075084 (263h23m33.125155215s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.716788163 (263h23m34.147868294s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.851890717 (263h23m34.282970848s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:40.293306092 (263h23m34.724386243s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:42.156929229 (263h23m36.58800936s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157075889 (263h23m36.58815601s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157126075 (263h23m36.588206206s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection Aug 19 16:49:42.157170951 (263h23m36.588251082s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: read tcp 188.166.225.170:34242->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:42.157215747 (263h23m36.588295878s) [seg_pos={0 0}, upload=273839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 19 16:49:42.385478364 (263h23m36.816558495s) [upload=273839]: Done uploading segments... Aug 19 16:49:42.385503091 (263h23m36.816583212s) [upload=273839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.041479465s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.948354295s FAIL (2025-08-19 17:16:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 263h18m33.75518017s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324344006 (263h18m33.755424127s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324633220 (263h18m33.755713351s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324851286 (263h18m33.755931417s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325068942 (263h18m33.756149063s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325283613 (263h18m33.756363744s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325499857 (263h18m33.756579998s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325759794 (263h18m33.756839915s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326059428 (263h18m33.757139549s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326277565 (263h18m33.757357696s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326470835 (263h18m33.757550966s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326673753 (263h18m33.757753884s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326867664 (263h18m33.757947795s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:40.851885389 (263h18m35.28296552s) [upload=273764]: Next returned nil segment Aug 19 16:44:40.851903334 (263h18m35.282983465s) [upload=273764]: Waiting for error group managing segments... Aug 19 16:44:41.244668372 (263h18m35.675748503s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.277280365 (263h18m35.708360496s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.328561665 (263h18m35.759641786s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.331747139 (263h18m35.76282727s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.357720320 (263h18m35.788800481s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.378190805 (263h18m35.809270936s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.486480333 (263h18m35.917560464s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:45:38.230570787 (263h19m32.661650918s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: expected piece hash; context deadline exceeded; EOF Aug 19 16:45:38.230979869 (263h19m32.66206s) [noise=true , address=77.90.31.18:4840 , piece_id=JWOETSFSZ3HELLZNLEASYNGW5OR64TMQVOFYYJAJVTHWCBHVTSLQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: read tcp 188.166.225.170:39946->77.90.31.18:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231017812 (263h19m32.662097943s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 188.166.225.170:44164->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231063489 (263h19m32.66214362s) [noise=true , address=45.91.116.14:4203 , piece_id=WRLBES3FI5EA5AQLZXJYGDROP3KBG3E2F22R4YD6A5KKI44PLV2Q, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 188.166.225.170:51738->45.91.116.14:4203: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231090952 (263h19m32.662171083s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: read tcp 188.166.225.170:57158->77.90.31.15:4707: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231110719 (263h19m32.66219085s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:50784->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231140165 (263h19m32.662220296s) [noise=true , address=77.90.31.16:4749 , piece_id=XCPXVBYLQH345FPIW4KDNLBFUCDDEW2H4B25WIXGOO5FVMAIVKPQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: read tcp 188.166.225.170:51668->77.90.31.16:4749: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231144814 (263h19m32.662224945s) [noise=true , address=77.90.31.19:4865 , piece_id=B3HUFIJZKVZJH4OBBO2DD7S3I7HX4LO2IQ74XJPO4DTW4QER35NA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1bguBghQonV5sTHaWteyUEnpp5KNzBKbvBpbvrUNFDCveHN77X, address:77.90.31.19:4865): protocol: read tcp 188.166.225.170:39398->77.90.31.19:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231177167 (263h19m32.662257298s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: read tcp 188.166.225.170:58526->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231184571 (263h19m32.662264702s) [noise=true , address=45.91.116.12:4236 , piece_id=LCRQS2GOSHOSLZBGQTGB3Q3PA4VN7KXNW66TDZ5O424TPJWTQE3A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH, address:45.91.116.12:4236): protocol: read tcp 188.166.225.170:57166->45.91.116.12:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231189770 (263h19m32.662269901s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 188.166.225.170:54242->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231215850 (263h19m32.662295981s) [seg_pos={0 0}, upload=273764]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 19 16:45:38.459100173 (263h19m32.890180304s) [upload=273764]: Done uploading segments... Aug 19 16:45:38.459128316 (263h19m32.890208447s) [upload=273764]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.948386848s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [X..........................................X................] duration: 1.832083118s FAIL (2025-08-19 17:16:30 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 17:04:06.343826144 (263h38m0.774906265s) [upload=274088]: Starting upload Aug 19 17:04:06.343875278 (263h38m0.774955409s) [upload=274088]: Done starting upload Aug 19 17:04:06.343971703 (263h38m0.775051834s) [upload=274088]: Uploading segments... Aug 19 17:04:06.344028321 (263h38m0.775108453s) [upload=274088]: Got next segment. Inline: true Aug 19 17:04:06.344033120 (263h38m0.775113252s) [upload=274088]: Waiting for error group managing segments... Aug 19 17:04:06.344068187 (263h38m0.775148318s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:04:06.344071964 (263h38m0.775152095s) [upload=274088]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:04:06.344074499 (263h38m0.77515462s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:04:06.653287654 (263h38m1.084367785s) [upload=274088]: Done uploading segments... Aug 19 17:04:06.653300519 (263h38m1.08438065s) [upload=274088]: Upload finished. err: <nil> Duration:* 1.8321106s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X....X...................................] duration: 6.121191162s FAIL (2025-08-19 17:16:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.121216692s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................X...X......................................] duration: 6.520057671s FAIL (2025-08-19 17:16:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.520079161s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X....XX....................................] duration: 5.71264086s FAIL (2025-08-19 17:16:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 19 16:49:12.801154313 (263h23m7.232234445s) [upload=273839]: Starting upload Aug 19 16:49:12.801215942 (263h23m7.232296073s) [upload=273839]: Done starting upload Aug 19 16:49:12.801330351 (263h23m7.232410482s) [upload=273839]: Uploading segments... Aug 19 16:49:12.801425372 (263h23m7.232505503s) [upload=273839]: Got next segment. Inline: false Aug 19 16:49:12.801455880 (263h23m7.232536012s) [upload=273839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:49:12.801461982 (263h23m7.232542113s) [upload=273839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:49:13.036708228 (263h23m7.467788359s) [seg_pos={0 0}, upload=273839]: Begin upload segment... Aug 19 16:49:13.036749197 (263h23m7.467829328s) [seg_pos={0 0}, upload=273839]: Done begin upload segment. Aug 19 16:49:13.036771971 (263h23m7.467852092s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.036780256 (263h23m7.467860437s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037135917 (263h23m7.468216048s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037326592 (263h23m7.468406723s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037403259 (263h23m7.46848339s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037667814 (263h23m7.468747945s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037761764 (263h23m7.468841896s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037943331 (263h23m7.469023462s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038070565 (263h23m7.469150696s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038168622 (263h23m7.469248753s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038300104 (263h23m7.469380235s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038396789 (263h23m7.46947692s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038622241 (263h23m7.469702362s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038679350 (263h23m7.469759481s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038854735 (263h23m7.469934856s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.039048045 (263h23m7.470128166s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:14.749064365 (263h23m9.180144496s) [upload=273839]: Next returned nil segment Aug 19 16:49:14.749117217 (263h23m9.180197348s) [upload=273839]: Waiting for error group managing segments... Aug 19 16:49:15.090317863 (263h23m9.521397994s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.132109007 (263h23m9.563189138s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.222090156 (263h23m9.653170287s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.328041780 (263h23m9.759121911s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.353998781 (263h23m9.785078912s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:36.709543777 (263h23m31.140623898s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:37.041062020 (263h23m31.472142141s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.183707791 (263h23m32.614787922s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.694075084 (263h23m33.125155215s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.716788163 (263h23m34.147868294s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.851890717 (263h23m34.282970848s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:40.293306092 (263h23m34.724386243s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:42.156929229 (263h23m36.58800936s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157075889 (263h23m36.58815601s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157126075 (263h23m36.588206206s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection Aug 19 16:49:42.157170951 (263h23m36.588251082s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: read tcp 188.166.225.170:34242->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:42.157215747 (263h23m36.588295878s) [seg_pos={0 0}, upload=273839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 19 16:49:42.385478364 (263h23m36.816558495s) [upload=273839]: Done uploading segments... Aug 19 16:49:42.385503091 (263h23m36.816583212s) [upload=273839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.712667851s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [X..............X......X.XX........................X.........] duration: 5.886717553s FAIL (2025-08-19 17:16:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== Aug 19 17:09:15.684096389 (263h43m10.11517652s) [upload=274176]: Starting upload Aug 19 17:09:15.684143008 (263h43m10.115223139s) [upload=274176]: Done starting upload Aug 19 17:09:15.684256185 (263h43m10.115336306s) [upload=274176]: Uploading segments... Aug 19 17:09:15.684468862 (263h43m10.115548993s) [upload=274176]: Got next segment. Inline: false Aug 19 17:09:15.684504950 (263h43m10.115585082s) [upload=274176]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:09:15.684509770 (263h43m10.115589901s) [upload=274176]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:09:16.135567491 (263h43m10.566647622s) [seg_pos={0 0}, upload=274176]: Begin upload segment... Aug 19 17:09:16.135613589 (263h43m10.56669372s) [seg_pos={0 0}, upload=274176]: Done begin upload segment. Aug 19 17:09:16.135636573 (263h43m10.566716695s) [noise=true , address=104.36.82.5:4853 , piece_id=K6TRSISY75NXYJNAAEALQ7W4RA26P5TGIVRMHX5SGOIF2GJ7X3GQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.135699574 (263h43m10.566779705s) [noise=true , address=104.36.82.60:4341 , piece_id=5NWVHAFJXPCBGPMRT42H4C3LPE3XKBDKPSX44D3DOWF6XXXJ5LFA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136028283 (263h43m10.567108404s) [noise=true , address=79.127.220.98:4432 , piece_id=4VA6KKCGA5GM55N3SD7FNUILD5XWYHE5VFRBFDIBAGVEK2VHQELA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136206604 (263h43m10.567286735s) [noise=true , address=104.36.82.22:4538 , piece_id=YUOOOPSZLQRNVU557APDO5FTOCY4FEYAHCBZQEE6YWMBPK5ZJYNA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136295745 (263h43m10.567375876s) [noise=true , address=77.90.31.11:4564 , piece_id=TX3C4VLEEOJOHBN7VDK4KIROG42PDDCB3SK26FZA7PM2455YZCMA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136640434 (263h43m10.567720565s) [noise=true , address=104.36.82.16:4315 , piece_id=P3SYICVJ5HSVY4MM62FDXQXJARZLIOQYNPBCT2SKYFPYZ3G3MSWQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136649702 (263h43m10.567729833s) [noise=true , address=104.36.82.44:4874 , piece_id=D3L2BM57MTGKIWKRSSEL7DRXUOK7LLKHL7M3Y3WAFCBILVDEV24Q, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136950888 (263h43m10.568031019s) [noise=true , address=104.36.82.35:4605 , piece_id=ORHH35ESP7KPGGDRD7ANECFHW7YTA2VFZTB5EMRCBQX5EWB7ZU4Q, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136961468 (263h43m10.568041599s) [noise=true , address=77.90.31.10:4535 , piece_id=N6LDVXERZAIR3EZ2TAMFQBM5Y565FMSP44I5KAWC32DTIOOM3FIQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137201027 (263h43m10.568281158s) [noise=true , address=77.90.31.12:4611 , piece_id=BANXKRHIMMOQFBSC4FW2E3SMNXCQERG2OTD7GXMM56T3X6JPNQGQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137260470 (263h43m10.568340601s) [noise=true , address=104.36.82.11:4134 , piece_id=BSGC4ADI6LMK3E2YEYPWFMYS3GC5VA42YM22VIJEGZ2JANYMLBYQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137463418 (263h43m10.568543549s) [noise=true , address=45.91.116.10:4110 , piece_id=KCZZOHPLSRNA45VXOMW7AKUYMRRFRUA52KSOBHNDRJPCB3RMV5HA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137553992 (263h43m10.568634123s) [noise=true , address=104.36.82.55:4224 , piece_id=PGZXUHGOONWDPIB4ZHAKLL4F2UWMZJPAL7BOTXIGSQWESLUPWGQA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137682447 (263h43m10.568762578s) [noise=true , address=45.91.116.12:4238 , piece_id=JAUKDCYIMCIOX4EFHQMCNCBLVZEIBKODPBL5QVBPOJIB554KLNRQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137883032 (263h43m10.568963163s) [noise=true , address=77.90.31.16:4743 , piece_id=YKPMLYPRMAZEBX7ORWEICH5AWKVS2J74D25W4ANIJJOUIBX5UBRA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137912418 (263h43m10.568992539s) [noise=true , address=104.36.82.23:4455 , piece_id=L2M5P5LRKVLVVDSYIOBV4SR3EAH2UO2JUZBGKZKIWBUIGFZPVEOA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.138136216 (263h43m10.569216347s) [noise=true , address=104.36.82.7:4930 , piece_id=EGTR52TZDAVTPPRCJLK4GOUXUWW2U7XJMQOV5P3JCKAKOK7GG5JQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.138374362 (263h43m10.569454493s) [noise=true , address=77.90.31.18:4839 , piece_id=BG553GCZSJEH5ORINSIQ7IUYU3UBQ3WO73ZAGRB7W45HOASJMBVQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:17.616571604 (263h43m12.047651765s) [upload=274176]: Next returned nil segment Aug 19 17:09:17.616591412 (263h43m12.047671543s) [upload=274176]: Waiting for error group managing segments... Aug 19 17:09:18.008239751 (263h43m12.439319882s) [noise=true , address=79.127.220.98:4432 , piece_id=4VA6KKCGA5GM55N3SD7FNUILD5XWYHE5VFRBFDIBAGVEK2VHQELA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.039911371 (263h43m12.470991502s) [noise=true , address=45.91.116.12:4238 , piece_id=JAUKDCYIMCIOX4EFHQMCNCBLVZEIBKODPBL5QVBPOJIB554KLNRQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.065695686 (263h43m12.496775817s) [noise=true , address=77.90.31.16:4743 , piece_id=YKPMLYPRMAZEBX7ORWEICH5AWKVS2J74D25W4ANIJJOUIBX5UBRA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.080085439 (263h43m12.51116557s) [noise=true , address=45.91.116.10:4110 , piece_id=KCZZOHPLSRNA45VXOMW7AKUYMRRFRUA52KSOBHNDRJPCB3RMV5HA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.131454157 (263h43m12.562534288s) [noise=true , address=104.36.82.11:4134 , piece_id=BSGC4ADI6LMK3E2YEYPWFMYS3GC5VA42YM22VIJEGZ2JANYMLBYQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.143557214 (263h43m12.574637345s) [noise=true , address=104.36.82.7:4930 , piece_id=EGTR52TZDAVTPPRCJLK4GOUXUWW2U7XJMQOV5P3JCKAKOK7GG5JQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.156135631 (263h43m12.587215762s) [noise=true , address=77.90.31.12:4611 , piece_id=BANXKRHIMMOQFBSC4FW2E3SMNXCQERG2OTD7GXMM56T3X6JPNQGQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.159345252 (263h43m12.590425373s) [noise=true , address=104.36.82.55:4224 , piece_id=PGZXUHGOONWDPIB4ZHAKLL4F2UWMZJPAL7BOTXIGSQWESLUPWGQA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.169106219 (263h43m12.60018635s) [noise=true , address=104.36.82.22:4538 , piece_id=YUOOOPSZLQRNVU557APDO5FTOCY4FEYAHCBZQEE6YWMBPK5ZJYNA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.188273593 (263h43m12.619353724s) [noise=true , address=104.36.82.35:4605 , piece_id=ORHH35ESP7KPGGDRD7ANECFHW7YTA2VFZTB5EMRCBQX5EWB7ZU4Q, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.204131846 (263h43m12.635211977s) [noise=true , address=104.36.82.16:4315 , piece_id=P3SYICVJ5HSVY4MM62FDXQXJARZLIOQYNPBCT2SKYFPYZ3G3MSWQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.217541965 (263h43m12.648622096s) [noise=true , address=104.36.82.60:4341 , piece_id=5NWVHAFJXPCBGPMRT42H4C3LPE3XKBDKPSX44D3DOWF6XXXJ5LFA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.222565827 (263h43m12.653645948s) [noise=true , address=104.36.82.23:4455 , piece_id=L2M5P5LRKVLVVDSYIOBV4SR3EAH2UO2JUZBGKZKIWBUIGFZPVEOA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.377619011 (263h43m12.808699132s) [noise=true , address=104.36.82.44:4874 , piece_id=D3L2BM57MTGKIWKRSSEL7DRXUOK7LLKHL7M3Y3WAFCBILVDEV24Q, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.377646604 (263h43m12.808726725s) [seg_pos={0 0}, upload=274176]: Segment reached optimal threshold of 14 pieces. Aug 19 17:09:18.377754100 (263h43m12.808834231s) [noise=true , address=104.36.82.5:4853 , piece_id=K6TRSISY75NXYJNAAEALQ7W4RA26P5TGIVRMHX5SGOIF2GJ7X3GQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377895570 (263h43m12.808975701s) [noise=true , address=77.90.31.18:4839 , piece_id=BG553GCZSJEH5ORINSIQ7IUYU3UBQ3WO73ZAGRB7W45HOASJMBVQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gmWwyaQ6zM6QdayBFpXdE9VSE1aaQcBHFcukJoASjjHWCDsNx): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377915529 (263h43m12.80899566s) [noise=true , address=77.90.31.11:4564 , piece_id=TX3C4VLEEOJOHBN7VDK4KIROG42PDDCB3SK26FZA7PM2455YZCMA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377974832 (263h43m12.809054963s) [noise=true , address=77.90.31.10:4535 , piece_id=N6LDVXERZAIR3EZ2TAMFQBM5Y565FMSP44I5KAWC32DTIOOM3FIQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377999008 (263h43m12.80907914s) [seg_pos={0 0}, upload=274176]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 17:09:18.378145599 (263h43m12.80922573s) [upload=274176]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 17:09:18.378150488 (263h43m12.809230619s) [upload=274176]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:09:18.983488776 (263h43m13.414568907s) [upload=274176]: Done uploading segments... Aug 19 17:09:18.983507222 (263h43m13.414587353s) [upload=274176]: Upload finished. err: <nil> Duration: 5.886746297s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.X..........................................X...............] duration: 1.87462699s FAIL (2025-08-19 17:15:52 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 17:04:06.343826144 (263h38m0.774906265s) [upload=274088]: Starting upload Aug 19 17:04:06.343875278 (263h38m0.774955409s) [upload=274088]: Done starting upload Aug 19 17:04:06.343971703 (263h38m0.775051834s) [upload=274088]: Uploading segments... Aug 19 17:04:06.344028321 (263h38m0.775108453s) [upload=274088]: Got next segment. Inline: true Aug 19 17:04:06.344033120 (263h38m0.775113252s) [upload=274088]: Waiting for error group managing segments... Aug 19 17:04:06.344068187 (263h38m0.775148318s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:04:06.344071964 (263h38m0.775152095s) [upload=274088]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:04:06.344074499 (263h38m0.77515462s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:04:06.653287654 (263h38m1.084367785s) [upload=274088]: Done uploading segments... Aug 19 17:04:06.653300519 (263h38m1.08438065s) [upload=274088]: Upload finished. err: <nil> Duration:* 1.874702244s core uplink us-select-1 us1-placement-20 metainfo-projects

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)