Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.948354295s FAIL (2025-08-19 17:16:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 263h18m33.75518017s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324344006 (263h18m33.755424127s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324633220 (263h18m33.755713351s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324851286 (263h18m33.755931417s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325068942 (263h18m33.756149063s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325283613 (263h18m33.756363744s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325499857 (263h18m33.756579998s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325759794 (263h18m33.756839915s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326059428 (263h18m33.757139549s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326277565 (263h18m33.757357696s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326470835 (263h18m33.757550966s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326673753 (263h18m33.757753884s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326867664 (263h18m33.757947795s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:40.851885389 (263h18m35.28296552s) [upload=273764]: Next returned nil segment Aug 19 16:44:40.851903334 (263h18m35.282983465s) [upload=273764]: Waiting for error group managing segments... Aug 19 16:44:41.244668372 (263h18m35.675748503s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.277280365 (263h18m35.708360496s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.328561665 (263h18m35.759641786s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.331747139 (263h18m35.76282727s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.357720320 (263h18m35.788800481s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.378190805 (263h18m35.809270936s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.486480333 (263h18m35.917560464s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:45:38.230570787 (263h19m32.661650918s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: expected piece hash; context deadline exceeded; EOF Aug 19 16:45:38.230979869 (263h19m32.66206s) [noise=true , address=77.90.31.18:4840 , piece_id=JWOETSFSZ3HELLZNLEASYNGW5OR64TMQVOFYYJAJVTHWCBHVTSLQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: read tcp 188.166.225.170:39946->77.90.31.18:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231017812 (263h19m32.662097943s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 188.166.225.170:44164->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231063489 (263h19m32.66214362s) [noise=true , address=45.91.116.14:4203 , piece_id=WRLBES3FI5EA5AQLZXJYGDROP3KBG3E2F22R4YD6A5KKI44PLV2Q, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 188.166.225.170:51738->45.91.116.14:4203: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231090952 (263h19m32.662171083s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: read tcp 188.166.225.170:57158->77.90.31.15:4707: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231110719 (263h19m32.66219085s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:50784->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231140165 (263h19m32.662220296s) [noise=true , address=77.90.31.16:4749 , piece_id=XCPXVBYLQH345FPIW4KDNLBFUCDDEW2H4B25WIXGOO5FVMAIVKPQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: read tcp 188.166.225.170:51668->77.90.31.16:4749: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231144814 (263h19m32.662224945s) [noise=true , address=77.90.31.19:4865 , piece_id=B3HUFIJZKVZJH4OBBO2DD7S3I7HX4LO2IQ74XJPO4DTW4QER35NA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1bguBghQonV5sTHaWteyUEnpp5KNzBKbvBpbvrUNFDCveHN77X, address:77.90.31.19:4865): protocol: read tcp 188.166.225.170:39398->77.90.31.19:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231177167 (263h19m32.662257298s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: read tcp 188.166.225.170:58526->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231184571 (263h19m32.662264702s) [noise=true , address=45.91.116.12:4236 , piece_id=LCRQS2GOSHOSLZBGQTGB3Q3PA4VN7KXNW66TDZ5O424TPJWTQE3A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH, address:45.91.116.12:4236): protocol: read tcp 188.166.225.170:57166->45.91.116.12:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231189770 (263h19m32.662269901s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 188.166.225.170:54242->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231215850 (263h19m32.662295981s) [seg_pos={0 0}, upload=273764]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 19 16:45:38.459100173 (263h19m32.890180304s) [upload=273764]: Done uploading segments... Aug 19 16:45:38.459128316 (263h19m32.890208447s) [upload=273764]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.948386848s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [X..........................................X................] duration: 1.832083118s FAIL (2025-08-19 17:16:30 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 17:04:06.343826144 (263h38m0.774906265s) [upload=274088]: Starting upload Aug 19 17:04:06.343875278 (263h38m0.774955409s) [upload=274088]: Done starting upload Aug 19 17:04:06.343971703 (263h38m0.775051834s) [upload=274088]: Uploading segments... Aug 19 17:04:06.344028321 (263h38m0.775108453s) [upload=274088]: Got next segment. Inline: true Aug 19 17:04:06.344033120 (263h38m0.775113252s) [upload=274088]: Waiting for error group managing segments... Aug 19 17:04:06.344068187 (263h38m0.775148318s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:04:06.344071964 (263h38m0.775152095s) [upload=274088]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:04:06.344074499 (263h38m0.77515462s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:04:06.653287654 (263h38m1.084367785s) [upload=274088]: Done uploading segments... Aug 19 17:04:06.653300519 (263h38m1.08438065s) [upload=274088]: Upload finished. err: <nil> Duration:* 1.8321106s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X....X...................................] duration: 6.121191162s FAIL (2025-08-19 17:16:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.121216692s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................X...X......................................] duration: 6.520057671s FAIL (2025-08-19 17:16:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.520079161s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X....XX....................................] duration: 5.71264086s FAIL (2025-08-19 17:16:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 19 16:49:12.801154313 (263h23m7.232234445s) [upload=273839]: Starting upload Aug 19 16:49:12.801215942 (263h23m7.232296073s) [upload=273839]: Done starting upload Aug 19 16:49:12.801330351 (263h23m7.232410482s) [upload=273839]: Uploading segments... Aug 19 16:49:12.801425372 (263h23m7.232505503s) [upload=273839]: Got next segment. Inline: false Aug 19 16:49:12.801455880 (263h23m7.232536012s) [upload=273839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:49:12.801461982 (263h23m7.232542113s) [upload=273839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:49:13.036708228 (263h23m7.467788359s) [seg_pos={0 0}, upload=273839]: Begin upload segment... Aug 19 16:49:13.036749197 (263h23m7.467829328s) [seg_pos={0 0}, upload=273839]: Done begin upload segment. Aug 19 16:49:13.036771971 (263h23m7.467852092s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.036780256 (263h23m7.467860437s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037135917 (263h23m7.468216048s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037326592 (263h23m7.468406723s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037403259 (263h23m7.46848339s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037667814 (263h23m7.468747945s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037761764 (263h23m7.468841896s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037943331 (263h23m7.469023462s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038070565 (263h23m7.469150696s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038168622 (263h23m7.469248753s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038300104 (263h23m7.469380235s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038396789 (263h23m7.46947692s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038622241 (263h23m7.469702362s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038679350 (263h23m7.469759481s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038854735 (263h23m7.469934856s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.039048045 (263h23m7.470128166s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:14.749064365 (263h23m9.180144496s) [upload=273839]: Next returned nil segment Aug 19 16:49:14.749117217 (263h23m9.180197348s) [upload=273839]: Waiting for error group managing segments... Aug 19 16:49:15.090317863 (263h23m9.521397994s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.132109007 (263h23m9.563189138s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.222090156 (263h23m9.653170287s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.328041780 (263h23m9.759121911s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.353998781 (263h23m9.785078912s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:36.709543777 (263h23m31.140623898s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:37.041062020 (263h23m31.472142141s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.183707791 (263h23m32.614787922s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.694075084 (263h23m33.125155215s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.716788163 (263h23m34.147868294s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.851890717 (263h23m34.282970848s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:40.293306092 (263h23m34.724386243s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:42.156929229 (263h23m36.58800936s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157075889 (263h23m36.58815601s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157126075 (263h23m36.588206206s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection Aug 19 16:49:42.157170951 (263h23m36.588251082s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: read tcp 188.166.225.170:34242->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:42.157215747 (263h23m36.588295878s) [seg_pos={0 0}, upload=273839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 19 16:49:42.385478364 (263h23m36.816558495s) [upload=273839]: Done uploading segments... Aug 19 16:49:42.385503091 (263h23m36.816583212s) [upload=273839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.712667851s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [X..............X......X.XX........................X.........] duration: 5.886717553s FAIL (2025-08-19 17:16:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== Aug 19 17:09:15.684096389 (263h43m10.11517652s) [upload=274176]: Starting upload Aug 19 17:09:15.684143008 (263h43m10.115223139s) [upload=274176]: Done starting upload Aug 19 17:09:15.684256185 (263h43m10.115336306s) [upload=274176]: Uploading segments... Aug 19 17:09:15.684468862 (263h43m10.115548993s) [upload=274176]: Got next segment. Inline: false Aug 19 17:09:15.684504950 (263h43m10.115585082s) [upload=274176]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:09:15.684509770 (263h43m10.115589901s) [upload=274176]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:09:16.135567491 (263h43m10.566647622s) [seg_pos={0 0}, upload=274176]: Begin upload segment... Aug 19 17:09:16.135613589 (263h43m10.56669372s) [seg_pos={0 0}, upload=274176]: Done begin upload segment. Aug 19 17:09:16.135636573 (263h43m10.566716695s) [noise=true , address=104.36.82.5:4853 , piece_id=K6TRSISY75NXYJNAAEALQ7W4RA26P5TGIVRMHX5SGOIF2GJ7X3GQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.135699574 (263h43m10.566779705s) [noise=true , address=104.36.82.60:4341 , piece_id=5NWVHAFJXPCBGPMRT42H4C3LPE3XKBDKPSX44D3DOWF6XXXJ5LFA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136028283 (263h43m10.567108404s) [noise=true , address=79.127.220.98:4432 , piece_id=4VA6KKCGA5GM55N3SD7FNUILD5XWYHE5VFRBFDIBAGVEK2VHQELA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136206604 (263h43m10.567286735s) [noise=true , address=104.36.82.22:4538 , piece_id=YUOOOPSZLQRNVU557APDO5FTOCY4FEYAHCBZQEE6YWMBPK5ZJYNA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136295745 (263h43m10.567375876s) [noise=true , address=77.90.31.11:4564 , piece_id=TX3C4VLEEOJOHBN7VDK4KIROG42PDDCB3SK26FZA7PM2455YZCMA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136640434 (263h43m10.567720565s) [noise=true , address=104.36.82.16:4315 , piece_id=P3SYICVJ5HSVY4MM62FDXQXJARZLIOQYNPBCT2SKYFPYZ3G3MSWQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136649702 (263h43m10.567729833s) [noise=true , address=104.36.82.44:4874 , piece_id=D3L2BM57MTGKIWKRSSEL7DRXUOK7LLKHL7M3Y3WAFCBILVDEV24Q, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136950888 (263h43m10.568031019s) [noise=true , address=104.36.82.35:4605 , piece_id=ORHH35ESP7KPGGDRD7ANECFHW7YTA2VFZTB5EMRCBQX5EWB7ZU4Q, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136961468 (263h43m10.568041599s) [noise=true , address=77.90.31.10:4535 , piece_id=N6LDVXERZAIR3EZ2TAMFQBM5Y565FMSP44I5KAWC32DTIOOM3FIQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137201027 (263h43m10.568281158s) [noise=true , address=77.90.31.12:4611 , piece_id=BANXKRHIMMOQFBSC4FW2E3SMNXCQERG2OTD7GXMM56T3X6JPNQGQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137260470 (263h43m10.568340601s) [noise=true , address=104.36.82.11:4134 , piece_id=BSGC4ADI6LMK3E2YEYPWFMYS3GC5VA42YM22VIJEGZ2JANYMLBYQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137463418 (263h43m10.568543549s) [noise=true , address=45.91.116.10:4110 , piece_id=KCZZOHPLSRNA45VXOMW7AKUYMRRFRUA52KSOBHNDRJPCB3RMV5HA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137553992 (263h43m10.568634123s) [noise=true , address=104.36.82.55:4224 , piece_id=PGZXUHGOONWDPIB4ZHAKLL4F2UWMZJPAL7BOTXIGSQWESLUPWGQA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137682447 (263h43m10.568762578s) [noise=true , address=45.91.116.12:4238 , piece_id=JAUKDCYIMCIOX4EFHQMCNCBLVZEIBKODPBL5QVBPOJIB554KLNRQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137883032 (263h43m10.568963163s) [noise=true , address=77.90.31.16:4743 , piece_id=YKPMLYPRMAZEBX7ORWEICH5AWKVS2J74D25W4ANIJJOUIBX5UBRA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137912418 (263h43m10.568992539s) [noise=true , address=104.36.82.23:4455 , piece_id=L2M5P5LRKVLVVDSYIOBV4SR3EAH2UO2JUZBGKZKIWBUIGFZPVEOA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.138136216 (263h43m10.569216347s) [noise=true , address=104.36.82.7:4930 , piece_id=EGTR52TZDAVTPPRCJLK4GOUXUWW2U7XJMQOV5P3JCKAKOK7GG5JQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.138374362 (263h43m10.569454493s) [noise=true , address=77.90.31.18:4839 , piece_id=BG553GCZSJEH5ORINSIQ7IUYU3UBQ3WO73ZAGRB7W45HOASJMBVQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:17.616571604 (263h43m12.047651765s) [upload=274176]: Next returned nil segment Aug 19 17:09:17.616591412 (263h43m12.047671543s) [upload=274176]: Waiting for error group managing segments... Aug 19 17:09:18.008239751 (263h43m12.439319882s) [noise=true , address=79.127.220.98:4432 , piece_id=4VA6KKCGA5GM55N3SD7FNUILD5XWYHE5VFRBFDIBAGVEK2VHQELA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.039911371 (263h43m12.470991502s) [noise=true , address=45.91.116.12:4238 , piece_id=JAUKDCYIMCIOX4EFHQMCNCBLVZEIBKODPBL5QVBPOJIB554KLNRQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.065695686 (263h43m12.496775817s) [noise=true , address=77.90.31.16:4743 , piece_id=YKPMLYPRMAZEBX7ORWEICH5AWKVS2J74D25W4ANIJJOUIBX5UBRA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.080085439 (263h43m12.51116557s) [noise=true , address=45.91.116.10:4110 , piece_id=KCZZOHPLSRNA45VXOMW7AKUYMRRFRUA52KSOBHNDRJPCB3RMV5HA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.131454157 (263h43m12.562534288s) [noise=true , address=104.36.82.11:4134 , piece_id=BSGC4ADI6LMK3E2YEYPWFMYS3GC5VA42YM22VIJEGZ2JANYMLBYQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.143557214 (263h43m12.574637345s) [noise=true , address=104.36.82.7:4930 , piece_id=EGTR52TZDAVTPPRCJLK4GOUXUWW2U7XJMQOV5P3JCKAKOK7GG5JQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.156135631 (263h43m12.587215762s) [noise=true , address=77.90.31.12:4611 , piece_id=BANXKRHIMMOQFBSC4FW2E3SMNXCQERG2OTD7GXMM56T3X6JPNQGQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.159345252 (263h43m12.590425373s) [noise=true , address=104.36.82.55:4224 , piece_id=PGZXUHGOONWDPIB4ZHAKLL4F2UWMZJPAL7BOTXIGSQWESLUPWGQA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.169106219 (263h43m12.60018635s) [noise=true , address=104.36.82.22:4538 , piece_id=YUOOOPSZLQRNVU557APDO5FTOCY4FEYAHCBZQEE6YWMBPK5ZJYNA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.188273593 (263h43m12.619353724s) [noise=true , address=104.36.82.35:4605 , piece_id=ORHH35ESP7KPGGDRD7ANECFHW7YTA2VFZTB5EMRCBQX5EWB7ZU4Q, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.204131846 (263h43m12.635211977s) [noise=true , address=104.36.82.16:4315 , piece_id=P3SYICVJ5HSVY4MM62FDXQXJARZLIOQYNPBCT2SKYFPYZ3G3MSWQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.217541965 (263h43m12.648622096s) [noise=true , address=104.36.82.60:4341 , piece_id=5NWVHAFJXPCBGPMRT42H4C3LPE3XKBDKPSX44D3DOWF6XXXJ5LFA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.222565827 (263h43m12.653645948s) [noise=true , address=104.36.82.23:4455 , piece_id=L2M5P5LRKVLVVDSYIOBV4SR3EAH2UO2JUZBGKZKIWBUIGFZPVEOA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.377619011 (263h43m12.808699132s) [noise=true , address=104.36.82.44:4874 , piece_id=D3L2BM57MTGKIWKRSSEL7DRXUOK7LLKHL7M3Y3WAFCBILVDEV24Q, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.377646604 (263h43m12.808726725s) [seg_pos={0 0}, upload=274176]: Segment reached optimal threshold of 14 pieces. Aug 19 17:09:18.377754100 (263h43m12.808834231s) [noise=true , address=104.36.82.5:4853 , piece_id=K6TRSISY75NXYJNAAEALQ7W4RA26P5TGIVRMHX5SGOIF2GJ7X3GQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377895570 (263h43m12.808975701s) [noise=true , address=77.90.31.18:4839 , piece_id=BG553GCZSJEH5ORINSIQ7IUYU3UBQ3WO73ZAGRB7W45HOASJMBVQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gmWwyaQ6zM6QdayBFpXdE9VSE1aaQcBHFcukJoASjjHWCDsNx): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377915529 (263h43m12.80899566s) [noise=true , address=77.90.31.11:4564 , piece_id=TX3C4VLEEOJOHBN7VDK4KIROG42PDDCB3SK26FZA7PM2455YZCMA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377974832 (263h43m12.809054963s) [noise=true , address=77.90.31.10:4535 , piece_id=N6LDVXERZAIR3EZ2TAMFQBM5Y565FMSP44I5KAWC32DTIOOM3FIQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377999008 (263h43m12.80907914s) [seg_pos={0 0}, upload=274176]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 17:09:18.378145599 (263h43m12.80922573s) [upload=274176]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 17:09:18.378150488 (263h43m12.809230619s) [upload=274176]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:09:18.983488776 (263h43m13.414568907s) [upload=274176]: Done uploading segments... Aug 19 17:09:18.983507222 (263h43m13.414587353s) [upload=274176]: Upload finished. err: <nil> Duration: 5.886746297s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.X..........................................X...............] duration: 1.87462699s FAIL (2025-08-19 17:15:52 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 17:04:06.343826144 (263h38m0.774906265s) [upload=274088]: Starting upload Aug 19 17:04:06.343875278 (263h38m0.774955409s) [upload=274088]: Done starting upload Aug 19 17:04:06.343971703 (263h38m0.775051834s) [upload=274088]: Uploading segments... Aug 19 17:04:06.344028321 (263h38m0.775108453s) [upload=274088]: Got next segment. Inline: true Aug 19 17:04:06.344033120 (263h38m0.775113252s) [upload=274088]: Waiting for error group managing segments... Aug 19 17:04:06.344068187 (263h38m0.775148318s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:04:06.344071964 (263h38m0.775152095s) [upload=274088]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:04:06.344074499 (263h38m0.77515462s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:04:06.653287654 (263h38m1.084367785s) [upload=274088]: Done uploading segments... Aug 19 17:04:06.653300519 (263h38m1.08438065s) [upload=274088]: Upload finished. err: <nil> Duration:* 1.874702244s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.733468265s FAIL (2025-08-19 17:15:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 263h18m33.75518017s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324344006 (263h18m33.755424127s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324633220 (263h18m33.755713351s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324851286 (263h18m33.755931417s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325068942 (263h18m33.756149063s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325283613 (263h18m33.756363744s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325499857 (263h18m33.756579998s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325759794 (263h18m33.756839915s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326059428 (263h18m33.757139549s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326277565 (263h18m33.757357696s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326470835 (263h18m33.757550966s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326673753 (263h18m33.757753884s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326867664 (263h18m33.757947795s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:40.851885389 (263h18m35.28296552s) [upload=273764]: Next returned nil segment Aug 19 16:44:40.851903334 (263h18m35.282983465s) [upload=273764]: Waiting for error group managing segments... Aug 19 16:44:41.244668372 (263h18m35.675748503s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.277280365 (263h18m35.708360496s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.328561665 (263h18m35.759641786s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.331747139 (263h18m35.76282727s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.357720320 (263h18m35.788800481s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.378190805 (263h18m35.809270936s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.486480333 (263h18m35.917560464s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:45:38.230570787 (263h19m32.661650918s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: expected piece hash; context deadline exceeded; EOF Aug 19 16:45:38.230979869 (263h19m32.66206s) [noise=true , address=77.90.31.18:4840 , piece_id=JWOETSFSZ3HELLZNLEASYNGW5OR64TMQVOFYYJAJVTHWCBHVTSLQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: read tcp 188.166.225.170:39946->77.90.31.18:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231017812 (263h19m32.662097943s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 188.166.225.170:44164->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231063489 (263h19m32.66214362s) [noise=true , address=45.91.116.14:4203 , piece_id=WRLBES3FI5EA5AQLZXJYGDROP3KBG3E2F22R4YD6A5KKI44PLV2Q, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 188.166.225.170:51738->45.91.116.14:4203: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231090952 (263h19m32.662171083s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: read tcp 188.166.225.170:57158->77.90.31.15:4707: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231110719 (263h19m32.66219085s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:50784->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231140165 (263h19m32.662220296s) [noise=true , address=77.90.31.16:4749 , piece_id=XCPXVBYLQH345FPIW4KDNLBFUCDDEW2H4B25WIXGOO5FVMAIVKPQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: read tcp 188.166.225.170:51668->77.90.31.16:4749: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231144814 (263h19m32.662224945s) [noise=true , address=77.90.31.19:4865 , piece_id=B3HUFIJZKVZJH4OBBO2DD7S3I7HX4LO2IQ74XJPO4DTW4QER35NA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1bguBghQonV5sTHaWteyUEnpp5KNzBKbvBpbvrUNFDCveHN77X, address:77.90.31.19:4865): protocol: read tcp 188.166.225.170:39398->77.90.31.19:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231177167 (263h19m32.662257298s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: read tcp 188.166.225.170:58526->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231184571 (263h19m32.662264702s) [noise=true , address=45.91.116.12:4236 , piece_id=LCRQS2GOSHOSLZBGQTGB3Q3PA4VN7KXNW66TDZ5O424TPJWTQE3A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH, address:45.91.116.12:4236): protocol: read tcp 188.166.225.170:57166->45.91.116.12:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231189770 (263h19m32.662269901s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 188.166.225.170:54242->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231215850 (263h19m32.662295981s) [seg_pos={0 0}, upload=273764]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 19 16:45:38.459100173 (263h19m32.890180304s) [upload=273764]: Done uploading segments... Aug 19 16:45:38.459128316 (263h19m32.890208447s) [upload=273764]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.733529292s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X....XX...................................] duration: 5.718191192s FAIL (2025-08-19 17:15:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 19 16:49:12.801154313 (263h23m7.232234445s) [upload=273839]: Starting upload Aug 19 16:49:12.801215942 (263h23m7.232296073s) [upload=273839]: Done starting upload Aug 19 16:49:12.801330351 (263h23m7.232410482s) [upload=273839]: Uploading segments... Aug 19 16:49:12.801425372 (263h23m7.232505503s) [upload=273839]: Got next segment. Inline: false Aug 19 16:49:12.801455880 (263h23m7.232536012s) [upload=273839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:49:12.801461982 (263h23m7.232542113s) [upload=273839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:49:13.036708228 (263h23m7.467788359s) [seg_pos={0 0}, upload=273839]: Begin upload segment... Aug 19 16:49:13.036749197 (263h23m7.467829328s) [seg_pos={0 0}, upload=273839]: Done begin upload segment. Aug 19 16:49:13.036771971 (263h23m7.467852092s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.036780256 (263h23m7.467860437s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037135917 (263h23m7.468216048s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037326592 (263h23m7.468406723s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037403259 (263h23m7.46848339s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037667814 (263h23m7.468747945s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037761764 (263h23m7.468841896s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037943331 (263h23m7.469023462s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038070565 (263h23m7.469150696s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038168622 (263h23m7.469248753s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038300104 (263h23m7.469380235s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038396789 (263h23m7.46947692s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038622241 (263h23m7.469702362s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038679350 (263h23m7.469759481s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038854735 (263h23m7.469934856s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.039048045 (263h23m7.470128166s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:14.749064365 (263h23m9.180144496s) [upload=273839]: Next returned nil segment Aug 19 16:49:14.749117217 (263h23m9.180197348s) [upload=273839]: Waiting for error group managing segments... Aug 19 16:49:15.090317863 (263h23m9.521397994s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.132109007 (263h23m9.563189138s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.222090156 (263h23m9.653170287s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.328041780 (263h23m9.759121911s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.353998781 (263h23m9.785078912s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:36.709543777 (263h23m31.140623898s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:37.041062020 (263h23m31.472142141s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.183707791 (263h23m32.614787922s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.694075084 (263h23m33.125155215s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.716788163 (263h23m34.147868294s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.851890717 (263h23m34.282970848s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:40.293306092 (263h23m34.724386243s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:42.156929229 (263h23m36.58800936s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157075889 (263h23m36.58815601s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157126075 (263h23m36.588206206s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection Aug 19 16:49:42.157170951 (263h23m36.588251082s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: read tcp 188.166.225.170:34242->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:42.157215747 (263h23m36.588295878s) [seg_pos={0 0}, upload=273839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 19 16:49:42.385478364 (263h23m36.816558495s) [upload=273839]: Done uploading segments... Aug 19 16:49:42.385503091 (263h23m36.816583212s) [upload=273839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.718223263s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X..............X......X.XX........................X........] duration: 5.735404196s FAIL (2025-08-19 17:15:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== Aug 19 17:09:15.684096389 (263h43m10.11517652s) [upload=274176]: Starting upload Aug 19 17:09:15.684143008 (263h43m10.115223139s) [upload=274176]: Done starting upload Aug 19 17:09:15.684256185 (263h43m10.115336306s) [upload=274176]: Uploading segments... Aug 19 17:09:15.684468862 (263h43m10.115548993s) [upload=274176]: Got next segment. Inline: false Aug 19 17:09:15.684504950 (263h43m10.115585082s) [upload=274176]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:09:15.684509770 (263h43m10.115589901s) [upload=274176]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:09:16.135567491 (263h43m10.566647622s) [seg_pos={0 0}, upload=274176]: Begin upload segment... Aug 19 17:09:16.135613589 (263h43m10.56669372s) [seg_pos={0 0}, upload=274176]: Done begin upload segment. Aug 19 17:09:16.135636573 (263h43m10.566716695s) [noise=true , address=104.36.82.5:4853 , piece_id=K6TRSISY75NXYJNAAEALQ7W4RA26P5TGIVRMHX5SGOIF2GJ7X3GQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.135699574 (263h43m10.566779705s) [noise=true , address=104.36.82.60:4341 , piece_id=5NWVHAFJXPCBGPMRT42H4C3LPE3XKBDKPSX44D3DOWF6XXXJ5LFA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136028283 (263h43m10.567108404s) [noise=true , address=79.127.220.98:4432 , piece_id=4VA6KKCGA5GM55N3SD7FNUILD5XWYHE5VFRBFDIBAGVEK2VHQELA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136206604 (263h43m10.567286735s) [noise=true , address=104.36.82.22:4538 , piece_id=YUOOOPSZLQRNVU557APDO5FTOCY4FEYAHCBZQEE6YWMBPK5ZJYNA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136295745 (263h43m10.567375876s) [noise=true , address=77.90.31.11:4564 , piece_id=TX3C4VLEEOJOHBN7VDK4KIROG42PDDCB3SK26FZA7PM2455YZCMA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136640434 (263h43m10.567720565s) [noise=true , address=104.36.82.16:4315 , piece_id=P3SYICVJ5HSVY4MM62FDXQXJARZLIOQYNPBCT2SKYFPYZ3G3MSWQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136649702 (263h43m10.567729833s) [noise=true , address=104.36.82.44:4874 , piece_id=D3L2BM57MTGKIWKRSSEL7DRXUOK7LLKHL7M3Y3WAFCBILVDEV24Q, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136950888 (263h43m10.568031019s) [noise=true , address=104.36.82.35:4605 , piece_id=ORHH35ESP7KPGGDRD7ANECFHW7YTA2VFZTB5EMRCBQX5EWB7ZU4Q, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136961468 (263h43m10.568041599s) [noise=true , address=77.90.31.10:4535 , piece_id=N6LDVXERZAIR3EZ2TAMFQBM5Y565FMSP44I5KAWC32DTIOOM3FIQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137201027 (263h43m10.568281158s) [noise=true , address=77.90.31.12:4611 , piece_id=BANXKRHIMMOQFBSC4FW2E3SMNXCQERG2OTD7GXMM56T3X6JPNQGQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137260470 (263h43m10.568340601s) [noise=true , address=104.36.82.11:4134 , piece_id=BSGC4ADI6LMK3E2YEYPWFMYS3GC5VA42YM22VIJEGZ2JANYMLBYQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137463418 (263h43m10.568543549s) [noise=true , address=45.91.116.10:4110 , piece_id=KCZZOHPLSRNA45VXOMW7AKUYMRRFRUA52KSOBHNDRJPCB3RMV5HA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137553992 (263h43m10.568634123s) [noise=true , address=104.36.82.55:4224 , piece_id=PGZXUHGOONWDPIB4ZHAKLL4F2UWMZJPAL7BOTXIGSQWESLUPWGQA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137682447 (263h43m10.568762578s) [noise=true , address=45.91.116.12:4238 , piece_id=JAUKDCYIMCIOX4EFHQMCNCBLVZEIBKODPBL5QVBPOJIB554KLNRQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137883032 (263h43m10.568963163s) [noise=true , address=77.90.31.16:4743 , piece_id=YKPMLYPRMAZEBX7ORWEICH5AWKVS2J74D25W4ANIJJOUIBX5UBRA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137912418 (263h43m10.568992539s) [noise=true , address=104.36.82.23:4455 , piece_id=L2M5P5LRKVLVVDSYIOBV4SR3EAH2UO2JUZBGKZKIWBUIGFZPVEOA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.138136216 (263h43m10.569216347s) [noise=true , address=104.36.82.7:4930 , piece_id=EGTR52TZDAVTPPRCJLK4GOUXUWW2U7XJMQOV5P3JCKAKOK7GG5JQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.138374362 (263h43m10.569454493s) [noise=true , address=77.90.31.18:4839 , piece_id=BG553GCZSJEH5ORINSIQ7IUYU3UBQ3WO73ZAGRB7W45HOASJMBVQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:17.616571604 (263h43m12.047651765s) [upload=274176]: Next returned nil segment Aug 19 17:09:17.616591412 (263h43m12.047671543s) [upload=274176]: Waiting for error group managing segments... Aug 19 17:09:18.008239751 (263h43m12.439319882s) [noise=true , address=79.127.220.98:4432 , piece_id=4VA6KKCGA5GM55N3SD7FNUILD5XWYHE5VFRBFDIBAGVEK2VHQELA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.039911371 (263h43m12.470991502s) [noise=true , address=45.91.116.12:4238 , piece_id=JAUKDCYIMCIOX4EFHQMCNCBLVZEIBKODPBL5QVBPOJIB554KLNRQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.065695686 (263h43m12.496775817s) [noise=true , address=77.90.31.16:4743 , piece_id=YKPMLYPRMAZEBX7ORWEICH5AWKVS2J74D25W4ANIJJOUIBX5UBRA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.080085439 (263h43m12.51116557s) [noise=true , address=45.91.116.10:4110 , piece_id=KCZZOHPLSRNA45VXOMW7AKUYMRRFRUA52KSOBHNDRJPCB3RMV5HA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.131454157 (263h43m12.562534288s) [noise=true , address=104.36.82.11:4134 , piece_id=BSGC4ADI6LMK3E2YEYPWFMYS3GC5VA42YM22VIJEGZ2JANYMLBYQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.143557214 (263h43m12.574637345s) [noise=true , address=104.36.82.7:4930 , piece_id=EGTR52TZDAVTPPRCJLK4GOUXUWW2U7XJMQOV5P3JCKAKOK7GG5JQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.156135631 (263h43m12.587215762s) [noise=true , address=77.90.31.12:4611 , piece_id=BANXKRHIMMOQFBSC4FW2E3SMNXCQERG2OTD7GXMM56T3X6JPNQGQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.159345252 (263h43m12.590425373s) [noise=true , address=104.36.82.55:4224 , piece_id=PGZXUHGOONWDPIB4ZHAKLL4F2UWMZJPAL7BOTXIGSQWESLUPWGQA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.169106219 (263h43m12.60018635s) [noise=true , address=104.36.82.22:4538 , piece_id=YUOOOPSZLQRNVU557APDO5FTOCY4FEYAHCBZQEE6YWMBPK5ZJYNA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.188273593 (263h43m12.619353724s) [noise=true , address=104.36.82.35:4605 , piece_id=ORHH35ESP7KPGGDRD7ANECFHW7YTA2VFZTB5EMRCBQX5EWB7ZU4Q, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.204131846 (263h43m12.635211977s) [noise=true , address=104.36.82.16:4315 , piece_id=P3SYICVJ5HSVY4MM62FDXQXJARZLIOQYNPBCT2SKYFPYZ3G3MSWQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.217541965 (263h43m12.648622096s) [noise=true , address=104.36.82.60:4341 , piece_id=5NWVHAFJXPCBGPMRT42H4C3LPE3XKBDKPSX44D3DOWF6XXXJ5LFA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.222565827 (263h43m12.653645948s) [noise=true , address=104.36.82.23:4455 , piece_id=L2M5P5LRKVLVVDSYIOBV4SR3EAH2UO2JUZBGKZKIWBUIGFZPVEOA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.377619011 (263h43m12.808699132s) [noise=true , address=104.36.82.44:4874 , piece_id=D3L2BM57MTGKIWKRSSEL7DRXUOK7LLKHL7M3Y3WAFCBILVDEV24Q, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.377646604 (263h43m12.808726725s) [seg_pos={0 0}, upload=274176]: Segment reached optimal threshold of 14 pieces. Aug 19 17:09:18.377754100 (263h43m12.808834231s) [noise=true , address=104.36.82.5:4853 , piece_id=K6TRSISY75NXYJNAAEALQ7W4RA26P5TGIVRMHX5SGOIF2GJ7X3GQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377895570 (263h43m12.808975701s) [noise=true , address=77.90.31.18:4839 , piece_id=BG553GCZSJEH5ORINSIQ7IUYU3UBQ3WO73ZAGRB7W45HOASJMBVQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gmWwyaQ6zM6QdayBFpXdE9VSE1aaQcBHFcukJoASjjHWCDsNx): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377915529 (263h43m12.80899566s) [noise=true , address=77.90.31.11:4564 , piece_id=TX3C4VLEEOJOHBN7VDK4KIROG42PDDCB3SK26FZA7PM2455YZCMA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377974832 (263h43m12.809054963s) [noise=true , address=77.90.31.10:4535 , piece_id=N6LDVXERZAIR3EZ2TAMFQBM5Y565FMSP44I5KAWC32DTIOOM3FIQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377999008 (263h43m12.80907914s) [seg_pos={0 0}, upload=274176]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 17:09:18.378145599 (263h43m12.80922573s) [upload=274176]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 17:09:18.378150488 (263h43m12.809230619s) [upload=274176]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:09:18.983488776 (263h43m13.414568907s) [upload=274176]: Done uploading segments... Aug 19 17:09:18.983507222 (263h43m13.414587353s) [upload=274176]: Upload finished. err: <nil> Duration: 5.735476695s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X....X..................................] duration: 5.975509037s FAIL (2025-08-19 17:15:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.975529345s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................X...X.....................................] duration: 7.005662477s FAIL (2025-08-19 17:15:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.005686321s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X....XX..................................] duration: 5.906746143s FAIL (2025-08-19 17:15:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 19 16:49:12.801154313 (263h23m7.232234445s) [upload=273839]: Starting upload Aug 19 16:49:12.801215942 (263h23m7.232296073s) [upload=273839]: Done starting upload Aug 19 16:49:12.801330351 (263h23m7.232410482s) [upload=273839]: Uploading segments... Aug 19 16:49:12.801425372 (263h23m7.232505503s) [upload=273839]: Got next segment. Inline: false Aug 19 16:49:12.801455880 (263h23m7.232536012s) [upload=273839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:49:12.801461982 (263h23m7.232542113s) [upload=273839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:49:13.036708228 (263h23m7.467788359s) [seg_pos={0 0}, upload=273839]: Begin upload segment... Aug 19 16:49:13.036749197 (263h23m7.467829328s) [seg_pos={0 0}, upload=273839]: Done begin upload segment. Aug 19 16:49:13.036771971 (263h23m7.467852092s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.036780256 (263h23m7.467860437s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037135917 (263h23m7.468216048s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037326592 (263h23m7.468406723s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037403259 (263h23m7.46848339s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037667814 (263h23m7.468747945s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037761764 (263h23m7.468841896s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037943331 (263h23m7.469023462s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038070565 (263h23m7.469150696s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038168622 (263h23m7.469248753s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038300104 (263h23m7.469380235s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038396789 (263h23m7.46947692s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038622241 (263h23m7.469702362s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038679350 (263h23m7.469759481s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038854735 (263h23m7.469934856s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.039048045 (263h23m7.470128166s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:14.749064365 (263h23m9.180144496s) [upload=273839]: Next returned nil segment Aug 19 16:49:14.749117217 (263h23m9.180197348s) [upload=273839]: Waiting for error group managing segments... Aug 19 16:49:15.090317863 (263h23m9.521397994s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.132109007 (263h23m9.563189138s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.222090156 (263h23m9.653170287s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.328041780 (263h23m9.759121911s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.353998781 (263h23m9.785078912s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:36.709543777 (263h23m31.140623898s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:37.041062020 (263h23m31.472142141s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.183707791 (263h23m32.614787922s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.694075084 (263h23m33.125155215s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.716788163 (263h23m34.147868294s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.851890717 (263h23m34.282970848s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:40.293306092 (263h23m34.724386243s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:42.156929229 (263h23m36.58800936s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157075889 (263h23m36.58815601s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157126075 (263h23m36.588206206s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection Aug 19 16:49:42.157170951 (263h23m36.588251082s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: read tcp 188.166.225.170:34242->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:42.157215747 (263h23m36.588295878s) [seg_pos={0 0}, upload=273839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 19 16:49:42.385478364 (263h23m36.816558495s) [upload=273839]: Done uploading segments... Aug 19 16:49:42.385503091 (263h23m36.816583212s) [upload=273839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.906772162s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..X..........................................X..............] duration: 2.005752442s FAIL (2025-08-19 17:15:18 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 17:04:06.343826144 (263h38m0.774906265s) [upload=274088]: Starting upload Aug 19 17:04:06.343875278 (263h38m0.774955409s) [upload=274088]: Done starting upload Aug 19 17:04:06.343971703 (263h38m0.775051834s) [upload=274088]: Uploading segments... Aug 19 17:04:06.344028321 (263h38m0.775108453s) [upload=274088]: Got next segment. Inline: true Aug 19 17:04:06.344033120 (263h38m0.775113252s) [upload=274088]: Waiting for error group managing segments... Aug 19 17:04:06.344068187 (263h38m0.775148318s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:04:06.344071964 (263h38m0.775152095s) [upload=274088]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:04:06.344074499 (263h38m0.77515462s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:04:06.653287654 (263h38m1.084367785s) [upload=274088]: Done uploading segments... Aug 19 17:04:06.653300519 (263h38m1.08438065s) [upload=274088]: Upload finished. err: <nil> Duration:* 2.005784202s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X....X.................................] duration: 5.844296146s FAIL (2025-08-19 17:15:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.844319341s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................X...X....................................] duration: 6.28044477s FAIL (2025-08-19 17:15:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.280498733s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.932479205s FAIL (2025-08-19 17:14:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 263h18m33.75518017s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324344006 (263h18m33.755424127s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324633220 (263h18m33.755713351s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324851286 (263h18m33.755931417s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325068942 (263h18m33.756149063s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325283613 (263h18m33.756363744s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325499857 (263h18m33.756579998s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325759794 (263h18m33.756839915s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326059428 (263h18m33.757139549s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326277565 (263h18m33.757357696s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326470835 (263h18m33.757550966s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326673753 (263h18m33.757753884s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326867664 (263h18m33.757947795s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:40.851885389 (263h18m35.28296552s) [upload=273764]: Next returned nil segment Aug 19 16:44:40.851903334 (263h18m35.282983465s) [upload=273764]: Waiting for error group managing segments... Aug 19 16:44:41.244668372 (263h18m35.675748503s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.277280365 (263h18m35.708360496s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.328561665 (263h18m35.759641786s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.331747139 (263h18m35.76282727s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.357720320 (263h18m35.788800481s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.378190805 (263h18m35.809270936s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.486480333 (263h18m35.917560464s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:45:38.230570787 (263h19m32.661650918s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: expected piece hash; context deadline exceeded; EOF Aug 19 16:45:38.230979869 (263h19m32.66206s) [noise=true , address=77.90.31.18:4840 , piece_id=JWOETSFSZ3HELLZNLEASYNGW5OR64TMQVOFYYJAJVTHWCBHVTSLQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: read tcp 188.166.225.170:39946->77.90.31.18:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231017812 (263h19m32.662097943s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 188.166.225.170:44164->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231063489 (263h19m32.66214362s) [noise=true , address=45.91.116.14:4203 , piece_id=WRLBES3FI5EA5AQLZXJYGDROP3KBG3E2F22R4YD6A5KKI44PLV2Q, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 188.166.225.170:51738->45.91.116.14:4203: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231090952 (263h19m32.662171083s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: read tcp 188.166.225.170:57158->77.90.31.15:4707: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231110719 (263h19m32.66219085s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:50784->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231140165 (263h19m32.662220296s) [noise=true , address=77.90.31.16:4749 , piece_id=XCPXVBYLQH345FPIW4KDNLBFUCDDEW2H4B25WIXGOO5FVMAIVKPQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: read tcp 188.166.225.170:51668->77.90.31.16:4749: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231144814 (263h19m32.662224945s) [noise=true , address=77.90.31.19:4865 , piece_id=B3HUFIJZKVZJH4OBBO2DD7S3I7HX4LO2IQ74XJPO4DTW4QER35NA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1bguBghQonV5sTHaWteyUEnpp5KNzBKbvBpbvrUNFDCveHN77X, address:77.90.31.19:4865): protocol: read tcp 188.166.225.170:39398->77.90.31.19:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231177167 (263h19m32.662257298s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: read tcp 188.166.225.170:58526->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231184571 (263h19m32.662264702s) [noise=true , address=45.91.116.12:4236 , piece_id=LCRQS2GOSHOSLZBGQTGB3Q3PA4VN7KXNW66TDZ5O424TPJWTQE3A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH, address:45.91.116.12:4236): protocol: read tcp 188.166.225.170:57166->45.91.116.12:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231189770 (263h19m32.662269901s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 188.166.225.170:54242->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231215850 (263h19m32.662295981s) [seg_pos={0 0}, upload=273764]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 19 16:45:38.459100173 (263h19m32.890180304s) [upload=273764]: Done uploading segments... Aug 19 16:45:38.459128316 (263h19m32.890208447s) [upload=273764]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.932499002s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X..............X......X.XX........................X.......] duration: 5.964953289s FAIL (2025-08-19 17:14:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== Aug 19 17:09:15.684096389 (263h43m10.11517652s) [upload=274176]: Starting upload Aug 19 17:09:15.684143008 (263h43m10.115223139s) [upload=274176]: Done starting upload Aug 19 17:09:15.684256185 (263h43m10.115336306s) [upload=274176]: Uploading segments... Aug 19 17:09:15.684468862 (263h43m10.115548993s) [upload=274176]: Got next segment. Inline: false Aug 19 17:09:15.684504950 (263h43m10.115585082s) [upload=274176]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:09:15.684509770 (263h43m10.115589901s) [upload=274176]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:09:16.135567491 (263h43m10.566647622s) [seg_pos={0 0}, upload=274176]: Begin upload segment... Aug 19 17:09:16.135613589 (263h43m10.56669372s) [seg_pos={0 0}, upload=274176]: Done begin upload segment. Aug 19 17:09:16.135636573 (263h43m10.566716695s) [noise=true , address=104.36.82.5:4853 , piece_id=K6TRSISY75NXYJNAAEALQ7W4RA26P5TGIVRMHX5SGOIF2GJ7X3GQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.135699574 (263h43m10.566779705s) [noise=true , address=104.36.82.60:4341 , piece_id=5NWVHAFJXPCBGPMRT42H4C3LPE3XKBDKPSX44D3DOWF6XXXJ5LFA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136028283 (263h43m10.567108404s) [noise=true , address=79.127.220.98:4432 , piece_id=4VA6KKCGA5GM55N3SD7FNUILD5XWYHE5VFRBFDIBAGVEK2VHQELA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136206604 (263h43m10.567286735s) [noise=true , address=104.36.82.22:4538 , piece_id=YUOOOPSZLQRNVU557APDO5FTOCY4FEYAHCBZQEE6YWMBPK5ZJYNA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136295745 (263h43m10.567375876s) [noise=true , address=77.90.31.11:4564 , piece_id=TX3C4VLEEOJOHBN7VDK4KIROG42PDDCB3SK26FZA7PM2455YZCMA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136640434 (263h43m10.567720565s) [noise=true , address=104.36.82.16:4315 , piece_id=P3SYICVJ5HSVY4MM62FDXQXJARZLIOQYNPBCT2SKYFPYZ3G3MSWQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136649702 (263h43m10.567729833s) [noise=true , address=104.36.82.44:4874 , piece_id=D3L2BM57MTGKIWKRSSEL7DRXUOK7LLKHL7M3Y3WAFCBILVDEV24Q, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136950888 (263h43m10.568031019s) [noise=true , address=104.36.82.35:4605 , piece_id=ORHH35ESP7KPGGDRD7ANECFHW7YTA2VFZTB5EMRCBQX5EWB7ZU4Q, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136961468 (263h43m10.568041599s) [noise=true , address=77.90.31.10:4535 , piece_id=N6LDVXERZAIR3EZ2TAMFQBM5Y565FMSP44I5KAWC32DTIOOM3FIQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137201027 (263h43m10.568281158s) [noise=true , address=77.90.31.12:4611 , piece_id=BANXKRHIMMOQFBSC4FW2E3SMNXCQERG2OTD7GXMM56T3X6JPNQGQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137260470 (263h43m10.568340601s) [noise=true , address=104.36.82.11:4134 , piece_id=BSGC4ADI6LMK3E2YEYPWFMYS3GC5VA42YM22VIJEGZ2JANYMLBYQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137463418 (263h43m10.568543549s) [noise=true , address=45.91.116.10:4110 , piece_id=KCZZOHPLSRNA45VXOMW7AKUYMRRFRUA52KSOBHNDRJPCB3RMV5HA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137553992 (263h43m10.568634123s) [noise=true , address=104.36.82.55:4224 , piece_id=PGZXUHGOONWDPIB4ZHAKLL4F2UWMZJPAL7BOTXIGSQWESLUPWGQA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137682447 (263h43m10.568762578s) [noise=true , address=45.91.116.12:4238 , piece_id=JAUKDCYIMCIOX4EFHQMCNCBLVZEIBKODPBL5QVBPOJIB554KLNRQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137883032 (263h43m10.568963163s) [noise=true , address=77.90.31.16:4743 , piece_id=YKPMLYPRMAZEBX7ORWEICH5AWKVS2J74D25W4ANIJJOUIBX5UBRA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137912418 (263h43m10.568992539s) [noise=true , address=104.36.82.23:4455 , piece_id=L2M5P5LRKVLVVDSYIOBV4SR3EAH2UO2JUZBGKZKIWBUIGFZPVEOA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.138136216 (263h43m10.569216347s) [noise=true , address=104.36.82.7:4930 , piece_id=EGTR52TZDAVTPPRCJLK4GOUXUWW2U7XJMQOV5P3JCKAKOK7GG5JQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.138374362 (263h43m10.569454493s) [noise=true , address=77.90.31.18:4839 , piece_id=BG553GCZSJEH5ORINSIQ7IUYU3UBQ3WO73ZAGRB7W45HOASJMBVQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:17.616571604 (263h43m12.047651765s) [upload=274176]: Next returned nil segment Aug 19 17:09:17.616591412 (263h43m12.047671543s) [upload=274176]: Waiting for error group managing segments... Aug 19 17:09:18.008239751 (263h43m12.439319882s) [noise=true , address=79.127.220.98:4432 , piece_id=4VA6KKCGA5GM55N3SD7FNUILD5XWYHE5VFRBFDIBAGVEK2VHQELA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.039911371 (263h43m12.470991502s) [noise=true , address=45.91.116.12:4238 , piece_id=JAUKDCYIMCIOX4EFHQMCNCBLVZEIBKODPBL5QVBPOJIB554KLNRQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.065695686 (263h43m12.496775817s) [noise=true , address=77.90.31.16:4743 , piece_id=YKPMLYPRMAZEBX7ORWEICH5AWKVS2J74D25W4ANIJJOUIBX5UBRA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.080085439 (263h43m12.51116557s) [noise=true , address=45.91.116.10:4110 , piece_id=KCZZOHPLSRNA45VXOMW7AKUYMRRFRUA52KSOBHNDRJPCB3RMV5HA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.131454157 (263h43m12.562534288s) [noise=true , address=104.36.82.11:4134 , piece_id=BSGC4ADI6LMK3E2YEYPWFMYS3GC5VA42YM22VIJEGZ2JANYMLBYQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.143557214 (263h43m12.574637345s) [noise=true , address=104.36.82.7:4930 , piece_id=EGTR52TZDAVTPPRCJLK4GOUXUWW2U7XJMQOV5P3JCKAKOK7GG5JQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.156135631 (263h43m12.587215762s) [noise=true , address=77.90.31.12:4611 , piece_id=BANXKRHIMMOQFBSC4FW2E3SMNXCQERG2OTD7GXMM56T3X6JPNQGQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.159345252 (263h43m12.590425373s) [noise=true , address=104.36.82.55:4224 , piece_id=PGZXUHGOONWDPIB4ZHAKLL4F2UWMZJPAL7BOTXIGSQWESLUPWGQA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.169106219 (263h43m12.60018635s) [noise=true , address=104.36.82.22:4538 , piece_id=YUOOOPSZLQRNVU557APDO5FTOCY4FEYAHCBZQEE6YWMBPK5ZJYNA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.188273593 (263h43m12.619353724s) [noise=true , address=104.36.82.35:4605 , piece_id=ORHH35ESP7KPGGDRD7ANECFHW7YTA2VFZTB5EMRCBQX5EWB7ZU4Q, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.204131846 (263h43m12.635211977s) [noise=true , address=104.36.82.16:4315 , piece_id=P3SYICVJ5HSVY4MM62FDXQXJARZLIOQYNPBCT2SKYFPYZ3G3MSWQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.217541965 (263h43m12.648622096s) [noise=true , address=104.36.82.60:4341 , piece_id=5NWVHAFJXPCBGPMRT42H4C3LPE3XKBDKPSX44D3DOWF6XXXJ5LFA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.222565827 (263h43m12.653645948s) [noise=true , address=104.36.82.23:4455 , piece_id=L2M5P5LRKVLVVDSYIOBV4SR3EAH2UO2JUZBGKZKIWBUIGFZPVEOA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.377619011 (263h43m12.808699132s) [noise=true , address=104.36.82.44:4874 , piece_id=D3L2BM57MTGKIWKRSSEL7DRXUOK7LLKHL7M3Y3WAFCBILVDEV24Q, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.377646604 (263h43m12.808726725s) [seg_pos={0 0}, upload=274176]: Segment reached optimal threshold of 14 pieces. Aug 19 17:09:18.377754100 (263h43m12.808834231s) [noise=true , address=104.36.82.5:4853 , piece_id=K6TRSISY75NXYJNAAEALQ7W4RA26P5TGIVRMHX5SGOIF2GJ7X3GQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377895570 (263h43m12.808975701s) [noise=true , address=77.90.31.18:4839 , piece_id=BG553GCZSJEH5ORINSIQ7IUYU3UBQ3WO73ZAGRB7W45HOASJMBVQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gmWwyaQ6zM6QdayBFpXdE9VSE1aaQcBHFcukJoASjjHWCDsNx): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377915529 (263h43m12.80899566s) [noise=true , address=77.90.31.11:4564 , piece_id=TX3C4VLEEOJOHBN7VDK4KIROG42PDDCB3SK26FZA7PM2455YZCMA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377974832 (263h43m12.809054963s) [noise=true , address=77.90.31.10:4535 , piece_id=N6LDVXERZAIR3EZ2TAMFQBM5Y565FMSP44I5KAWC32DTIOOM3FIQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377999008 (263h43m12.80907914s) [seg_pos={0 0}, upload=274176]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 17:09:18.378145599 (263h43m12.80922573s) [upload=274176]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 17:09:18.378150488 (263h43m12.809230619s) [upload=274176]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:09:18.983488776 (263h43m13.414568907s) [upload=274176]: Done uploading segments... Aug 19 17:09:18.983507222 (263h43m13.414587353s) [upload=274176]: Upload finished. err: <nil> Duration: 5.964981222s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...X..........................................X.............] duration: 2.147185565s FAIL (2025-08-19 17:14:37 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 17:04:06.343826144 (263h38m0.774906265s) [upload=274088]: Starting upload Aug 19 17:04:06.343875278 (263h38m0.774955409s) [upload=274088]: Done starting upload Aug 19 17:04:06.343971703 (263h38m0.775051834s) [upload=274088]: Uploading segments... Aug 19 17:04:06.344028321 (263h38m0.775108453s) [upload=274088]: Got next segment. Inline: true Aug 19 17:04:06.344033120 (263h38m0.775113252s) [upload=274088]: Waiting for error group managing segments... Aug 19 17:04:06.344068187 (263h38m0.775148318s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:04:06.344071964 (263h38m0.775152095s) [upload=274088]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:04:06.344074499 (263h38m0.77515462s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:04:06.653287654 (263h38m1.084367785s) [upload=274088]: Done uploading segments... Aug 19 17:04:06.653300519 (263h38m1.08438065s) [upload=274088]: Upload finished. err: <nil> Duration:* 2.147212096s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X....XX.................................] duration: 5.90199226s FAIL (2025-08-19 17:14:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 19 16:49:12.801154313 (263h23m7.232234445s) [upload=273839]: Starting upload Aug 19 16:49:12.801215942 (263h23m7.232296073s) [upload=273839]: Done starting upload Aug 19 16:49:12.801330351 (263h23m7.232410482s) [upload=273839]: Uploading segments... Aug 19 16:49:12.801425372 (263h23m7.232505503s) [upload=273839]: Got next segment. Inline: false Aug 19 16:49:12.801455880 (263h23m7.232536012s) [upload=273839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:49:12.801461982 (263h23m7.232542113s) [upload=273839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:49:13.036708228 (263h23m7.467788359s) [seg_pos={0 0}, upload=273839]: Begin upload segment... Aug 19 16:49:13.036749197 (263h23m7.467829328s) [seg_pos={0 0}, upload=273839]: Done begin upload segment. Aug 19 16:49:13.036771971 (263h23m7.467852092s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.036780256 (263h23m7.467860437s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037135917 (263h23m7.468216048s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037326592 (263h23m7.468406723s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037403259 (263h23m7.46848339s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037667814 (263h23m7.468747945s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037761764 (263h23m7.468841896s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037943331 (263h23m7.469023462s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038070565 (263h23m7.469150696s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038168622 (263h23m7.469248753s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038300104 (263h23m7.469380235s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038396789 (263h23m7.46947692s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038622241 (263h23m7.469702362s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038679350 (263h23m7.469759481s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038854735 (263h23m7.469934856s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.039048045 (263h23m7.470128166s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:14.749064365 (263h23m9.180144496s) [upload=273839]: Next returned nil segment Aug 19 16:49:14.749117217 (263h23m9.180197348s) [upload=273839]: Waiting for error group managing segments... Aug 19 16:49:15.090317863 (263h23m9.521397994s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.132109007 (263h23m9.563189138s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.222090156 (263h23m9.653170287s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.328041780 (263h23m9.759121911s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.353998781 (263h23m9.785078912s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:36.709543777 (263h23m31.140623898s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:37.041062020 (263h23m31.472142141s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.183707791 (263h23m32.614787922s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.694075084 (263h23m33.125155215s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.716788163 (263h23m34.147868294s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.851890717 (263h23m34.282970848s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:40.293306092 (263h23m34.724386243s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:42.156929229 (263h23m36.58800936s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157075889 (263h23m36.58815601s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157126075 (263h23m36.588206206s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection Aug 19 16:49:42.157170951 (263h23m36.588251082s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: read tcp 188.166.225.170:34242->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:42.157215747 (263h23m36.588295878s) [seg_pos={0 0}, upload=273839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 19 16:49:42.385478364 (263h23m36.816558495s) [upload=273839]: Done uploading segments... Aug 19 16:49:42.385503091 (263h23m36.816583212s) [upload=273839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.902013441s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................X...X...................................] duration: 6.638938514s FAIL (2025-08-19 17:14:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.638959875s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X..............X......X.XX........................X......] duration: 6.058333236s FAIL (2025-08-19 17:14:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== Aug 19 17:09:15.684096389 (263h43m10.11517652s) [upload=274176]: Starting upload Aug 19 17:09:15.684143008 (263h43m10.115223139s) [upload=274176]: Done starting upload Aug 19 17:09:15.684256185 (263h43m10.115336306s) [upload=274176]: Uploading segments... Aug 19 17:09:15.684468862 (263h43m10.115548993s) [upload=274176]: Got next segment. Inline: false Aug 19 17:09:15.684504950 (263h43m10.115585082s) [upload=274176]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:09:15.684509770 (263h43m10.115589901s) [upload=274176]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:09:16.135567491 (263h43m10.566647622s) [seg_pos={0 0}, upload=274176]: Begin upload segment... Aug 19 17:09:16.135613589 (263h43m10.56669372s) [seg_pos={0 0}, upload=274176]: Done begin upload segment. Aug 19 17:09:16.135636573 (263h43m10.566716695s) [noise=true , address=104.36.82.5:4853 , piece_id=K6TRSISY75NXYJNAAEALQ7W4RA26P5TGIVRMHX5SGOIF2GJ7X3GQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.135699574 (263h43m10.566779705s) [noise=true , address=104.36.82.60:4341 , piece_id=5NWVHAFJXPCBGPMRT42H4C3LPE3XKBDKPSX44D3DOWF6XXXJ5LFA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136028283 (263h43m10.567108404s) [noise=true , address=79.127.220.98:4432 , piece_id=4VA6KKCGA5GM55N3SD7FNUILD5XWYHE5VFRBFDIBAGVEK2VHQELA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136206604 (263h43m10.567286735s) [noise=true , address=104.36.82.22:4538 , piece_id=YUOOOPSZLQRNVU557APDO5FTOCY4FEYAHCBZQEE6YWMBPK5ZJYNA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136295745 (263h43m10.567375876s) [noise=true , address=77.90.31.11:4564 , piece_id=TX3C4VLEEOJOHBN7VDK4KIROG42PDDCB3SK26FZA7PM2455YZCMA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136640434 (263h43m10.567720565s) [noise=true , address=104.36.82.16:4315 , piece_id=P3SYICVJ5HSVY4MM62FDXQXJARZLIOQYNPBCT2SKYFPYZ3G3MSWQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136649702 (263h43m10.567729833s) [noise=true , address=104.36.82.44:4874 , piece_id=D3L2BM57MTGKIWKRSSEL7DRXUOK7LLKHL7M3Y3WAFCBILVDEV24Q, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136950888 (263h43m10.568031019s) [noise=true , address=104.36.82.35:4605 , piece_id=ORHH35ESP7KPGGDRD7ANECFHW7YTA2VFZTB5EMRCBQX5EWB7ZU4Q, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136961468 (263h43m10.568041599s) [noise=true , address=77.90.31.10:4535 , piece_id=N6LDVXERZAIR3EZ2TAMFQBM5Y565FMSP44I5KAWC32DTIOOM3FIQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137201027 (263h43m10.568281158s) [noise=true , address=77.90.31.12:4611 , piece_id=BANXKRHIMMOQFBSC4FW2E3SMNXCQERG2OTD7GXMM56T3X6JPNQGQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137260470 (263h43m10.568340601s) [noise=true , address=104.36.82.11:4134 , piece_id=BSGC4ADI6LMK3E2YEYPWFMYS3GC5VA42YM22VIJEGZ2JANYMLBYQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137463418 (263h43m10.568543549s) [noise=true , address=45.91.116.10:4110 , piece_id=KCZZOHPLSRNA45VXOMW7AKUYMRRFRUA52KSOBHNDRJPCB3RMV5HA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137553992 (263h43m10.568634123s) [noise=true , address=104.36.82.55:4224 , piece_id=PGZXUHGOONWDPIB4ZHAKLL4F2UWMZJPAL7BOTXIGSQWESLUPWGQA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137682447 (263h43m10.568762578s) [noise=true , address=45.91.116.12:4238 , piece_id=JAUKDCYIMCIOX4EFHQMCNCBLVZEIBKODPBL5QVBPOJIB554KLNRQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137883032 (263h43m10.568963163s) [noise=true , address=77.90.31.16:4743 , piece_id=YKPMLYPRMAZEBX7ORWEICH5AWKVS2J74D25W4ANIJJOUIBX5UBRA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137912418 (263h43m10.568992539s) [noise=true , address=104.36.82.23:4455 , piece_id=L2M5P5LRKVLVVDSYIOBV4SR3EAH2UO2JUZBGKZKIWBUIGFZPVEOA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.138136216 (263h43m10.569216347s) [noise=true , address=104.36.82.7:4930 , piece_id=EGTR52TZDAVTPPRCJLK4GOUXUWW2U7XJMQOV5P3JCKAKOK7GG5JQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.138374362 (263h43m10.569454493s) [noise=true , address=77.90.31.18:4839 , piece_id=BG553GCZSJEH5ORINSIQ7IUYU3UBQ3WO73ZAGRB7W45HOASJMBVQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:17.616571604 (263h43m12.047651765s) [upload=274176]: Next returned nil segment Aug 19 17:09:17.616591412 (263h43m12.047671543s) [upload=274176]: Waiting for error group managing segments... Aug 19 17:09:18.008239751 (263h43m12.439319882s) [noise=true , address=79.127.220.98:4432 , piece_id=4VA6KKCGA5GM55N3SD7FNUILD5XWYHE5VFRBFDIBAGVEK2VHQELA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.039911371 (263h43m12.470991502s) [noise=true , address=45.91.116.12:4238 , piece_id=JAUKDCYIMCIOX4EFHQMCNCBLVZEIBKODPBL5QVBPOJIB554KLNRQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.065695686 (263h43m12.496775817s) [noise=true , address=77.90.31.16:4743 , piece_id=YKPMLYPRMAZEBX7ORWEICH5AWKVS2J74D25W4ANIJJOUIBX5UBRA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.080085439 (263h43m12.51116557s) [noise=true , address=45.91.116.10:4110 , piece_id=KCZZOHPLSRNA45VXOMW7AKUYMRRFRUA52KSOBHNDRJPCB3RMV5HA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.131454157 (263h43m12.562534288s) [noise=true , address=104.36.82.11:4134 , piece_id=BSGC4ADI6LMK3E2YEYPWFMYS3GC5VA42YM22VIJEGZ2JANYMLBYQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.143557214 (263h43m12.574637345s) [noise=true , address=104.36.82.7:4930 , piece_id=EGTR52TZDAVTPPRCJLK4GOUXUWW2U7XJMQOV5P3JCKAKOK7GG5JQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.156135631 (263h43m12.587215762s) [noise=true , address=77.90.31.12:4611 , piece_id=BANXKRHIMMOQFBSC4FW2E3SMNXCQERG2OTD7GXMM56T3X6JPNQGQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.159345252 (263h43m12.590425373s) [noise=true , address=104.36.82.55:4224 , piece_id=PGZXUHGOONWDPIB4ZHAKLL4F2UWMZJPAL7BOTXIGSQWESLUPWGQA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.169106219 (263h43m12.60018635s) [noise=true , address=104.36.82.22:4538 , piece_id=YUOOOPSZLQRNVU557APDO5FTOCY4FEYAHCBZQEE6YWMBPK5ZJYNA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.188273593 (263h43m12.619353724s) [noise=true , address=104.36.82.35:4605 , piece_id=ORHH35ESP7KPGGDRD7ANECFHW7YTA2VFZTB5EMRCBQX5EWB7ZU4Q, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.204131846 (263h43m12.635211977s) [noise=true , address=104.36.82.16:4315 , piece_id=P3SYICVJ5HSVY4MM62FDXQXJARZLIOQYNPBCT2SKYFPYZ3G3MSWQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.217541965 (263h43m12.648622096s) [noise=true , address=104.36.82.60:4341 , piece_id=5NWVHAFJXPCBGPMRT42H4C3LPE3XKBDKPSX44D3DOWF6XXXJ5LFA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.222565827 (263h43m12.653645948s) [noise=true , address=104.36.82.23:4455 , piece_id=L2M5P5LRKVLVVDSYIOBV4SR3EAH2UO2JUZBGKZKIWBUIGFZPVEOA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.377619011 (263h43m12.808699132s) [noise=true , address=104.36.82.44:4874 , piece_id=D3L2BM57MTGKIWKRSSEL7DRXUOK7LLKHL7M3Y3WAFCBILVDEV24Q, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.377646604 (263h43m12.808726725s) [seg_pos={0 0}, upload=274176]: Segment reached optimal threshold of 14 pieces. Aug 19 17:09:18.377754100 (263h43m12.808834231s) [noise=true , address=104.36.82.5:4853 , piece_id=K6TRSISY75NXYJNAAEALQ7W4RA26P5TGIVRMHX5SGOIF2GJ7X3GQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377895570 (263h43m12.808975701s) [noise=true , address=77.90.31.18:4839 , piece_id=BG553GCZSJEH5ORINSIQ7IUYU3UBQ3WO73ZAGRB7W45HOASJMBVQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gmWwyaQ6zM6QdayBFpXdE9VSE1aaQcBHFcukJoASjjHWCDsNx): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377915529 (263h43m12.80899566s) [noise=true , address=77.90.31.11:4564 , piece_id=TX3C4VLEEOJOHBN7VDK4KIROG42PDDCB3SK26FZA7PM2455YZCMA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377974832 (263h43m12.809054963s) [noise=true , address=77.90.31.10:4535 , piece_id=N6LDVXERZAIR3EZ2TAMFQBM5Y565FMSP44I5KAWC32DTIOOM3FIQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377999008 (263h43m12.80907914s) [seg_pos={0 0}, upload=274176]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 17:09:18.378145599 (263h43m12.80922573s) [upload=274176]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 17:09:18.378150488 (263h43m12.809230619s) [upload=274176]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:09:18.983488776 (263h43m13.414568907s) [upload=274176]: Done uploading segments... Aug 19 17:09:18.983507222 (263h43m13.414587353s) [upload=274176]: Upload finished. err: <nil> Duration: 6.058357461s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X....X................................] duration: 6.012378421s FAIL (2025-08-19 17:13:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.012400514s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.777883474s FAIL (2025-08-19 17:13:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 263h18m33.75518017s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324344006 (263h18m33.755424127s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324633220 (263h18m33.755713351s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324851286 (263h18m33.755931417s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325068942 (263h18m33.756149063s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325283613 (263h18m33.756363744s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325499857 (263h18m33.756579998s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325759794 (263h18m33.756839915s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326059428 (263h18m33.757139549s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326277565 (263h18m33.757357696s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326470835 (263h18m33.757550966s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326673753 (263h18m33.757753884s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326867664 (263h18m33.757947795s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:40.851885389 (263h18m35.28296552s) [upload=273764]: Next returned nil segment Aug 19 16:44:40.851903334 (263h18m35.282983465s) [upload=273764]: Waiting for error group managing segments... Aug 19 16:44:41.244668372 (263h18m35.675748503s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.277280365 (263h18m35.708360496s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.328561665 (263h18m35.759641786s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.331747139 (263h18m35.76282727s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.357720320 (263h18m35.788800481s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.378190805 (263h18m35.809270936s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.486480333 (263h18m35.917560464s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:45:38.230570787 (263h19m32.661650918s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: expected piece hash; context deadline exceeded; EOF Aug 19 16:45:38.230979869 (263h19m32.66206s) [noise=true , address=77.90.31.18:4840 , piece_id=JWOETSFSZ3HELLZNLEASYNGW5OR64TMQVOFYYJAJVTHWCBHVTSLQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: read tcp 188.166.225.170:39946->77.90.31.18:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231017812 (263h19m32.662097943s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 188.166.225.170:44164->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231063489 (263h19m32.66214362s) [noise=true , address=45.91.116.14:4203 , piece_id=WRLBES3FI5EA5AQLZXJYGDROP3KBG3E2F22R4YD6A5KKI44PLV2Q, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 188.166.225.170:51738->45.91.116.14:4203: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231090952 (263h19m32.662171083s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: read tcp 188.166.225.170:57158->77.90.31.15:4707: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231110719 (263h19m32.66219085s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:50784->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231140165 (263h19m32.662220296s) [noise=true , address=77.90.31.16:4749 , piece_id=XCPXVBYLQH345FPIW4KDNLBFUCDDEW2H4B25WIXGOO5FVMAIVKPQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: read tcp 188.166.225.170:51668->77.90.31.16:4749: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231144814 (263h19m32.662224945s) [noise=true , address=77.90.31.19:4865 , piece_id=B3HUFIJZKVZJH4OBBO2DD7S3I7HX4LO2IQ74XJPO4DTW4QER35NA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1bguBghQonV5sTHaWteyUEnpp5KNzBKbvBpbvrUNFDCveHN77X, address:77.90.31.19:4865): protocol: read tcp 188.166.225.170:39398->77.90.31.19:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231177167 (263h19m32.662257298s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: read tcp 188.166.225.170:58526->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231184571 (263h19m32.662264702s) [noise=true , address=45.91.116.12:4236 , piece_id=LCRQS2GOSHOSLZBGQTGB3Q3PA4VN7KXNW66TDZ5O424TPJWTQE3A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH, address:45.91.116.12:4236): protocol: read tcp 188.166.225.170:57166->45.91.116.12:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231189770 (263h19m32.662269901s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 188.166.225.170:54242->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231215850 (263h19m32.662295981s) [seg_pos={0 0}, upload=273764]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 19 16:45:38.459100173 (263h19m32.890180304s) [upload=273764]: Done uploading segments... Aug 19 16:45:38.459128316 (263h19m32.890208447s) [upload=273764]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.777916107s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X....XX................................] duration: 6.002445255s FAIL (2025-08-19 17:13:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 19 16:49:12.801154313 (263h23m7.232234445s) [upload=273839]: Starting upload Aug 19 16:49:12.801215942 (263h23m7.232296073s) [upload=273839]: Done starting upload Aug 19 16:49:12.801330351 (263h23m7.232410482s) [upload=273839]: Uploading segments... Aug 19 16:49:12.801425372 (263h23m7.232505503s) [upload=273839]: Got next segment. Inline: false Aug 19 16:49:12.801455880 (263h23m7.232536012s) [upload=273839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:49:12.801461982 (263h23m7.232542113s) [upload=273839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:49:13.036708228 (263h23m7.467788359s) [seg_pos={0 0}, upload=273839]: Begin upload segment... Aug 19 16:49:13.036749197 (263h23m7.467829328s) [seg_pos={0 0}, upload=273839]: Done begin upload segment. Aug 19 16:49:13.036771971 (263h23m7.467852092s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.036780256 (263h23m7.467860437s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037135917 (263h23m7.468216048s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037326592 (263h23m7.468406723s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037403259 (263h23m7.46848339s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037667814 (263h23m7.468747945s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037761764 (263h23m7.468841896s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037943331 (263h23m7.469023462s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038070565 (263h23m7.469150696s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038168622 (263h23m7.469248753s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038300104 (263h23m7.469380235s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038396789 (263h23m7.46947692s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038622241 (263h23m7.469702362s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038679350 (263h23m7.469759481s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038854735 (263h23m7.469934856s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.039048045 (263h23m7.470128166s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:14.749064365 (263h23m9.180144496s) [upload=273839]: Next returned nil segment Aug 19 16:49:14.749117217 (263h23m9.180197348s) [upload=273839]: Waiting for error group managing segments... Aug 19 16:49:15.090317863 (263h23m9.521397994s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.132109007 (263h23m9.563189138s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.222090156 (263h23m9.653170287s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.328041780 (263h23m9.759121911s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.353998781 (263h23m9.785078912s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:36.709543777 (263h23m31.140623898s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:37.041062020 (263h23m31.472142141s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.183707791 (263h23m32.614787922s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.694075084 (263h23m33.125155215s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.716788163 (263h23m34.147868294s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.851890717 (263h23m34.282970848s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:40.293306092 (263h23m34.724386243s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:42.156929229 (263h23m36.58800936s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157075889 (263h23m36.58815601s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157126075 (263h23m36.588206206s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection Aug 19 16:49:42.157170951 (263h23m36.588251082s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: read tcp 188.166.225.170:34242->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:42.157215747 (263h23m36.588295878s) [seg_pos={0 0}, upload=273839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 19 16:49:42.385478364 (263h23m36.816558495s) [upload=273839]: Done uploading segments... Aug 19 16:49:42.385503091 (263h23m36.816583212s) [upload=273839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.002471445s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [X...X..............X......X.XX........................X.....] duration: 5.546270581s FAIL (2025-08-19 17:13:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== Aug 19 17:09:15.684096389 (263h43m10.11517652s) [upload=274176]: Starting upload Aug 19 17:09:15.684143008 (263h43m10.115223139s) [upload=274176]: Done starting upload Aug 19 17:09:15.684256185 (263h43m10.115336306s) [upload=274176]: Uploading segments... Aug 19 17:09:15.684468862 (263h43m10.115548993s) [upload=274176]: Got next segment. Inline: false Aug 19 17:09:15.684504950 (263h43m10.115585082s) [upload=274176]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:09:15.684509770 (263h43m10.115589901s) [upload=274176]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:09:16.135567491 (263h43m10.566647622s) [seg_pos={0 0}, upload=274176]: Begin upload segment... Aug 19 17:09:16.135613589 (263h43m10.56669372s) [seg_pos={0 0}, upload=274176]: Done begin upload segment. Aug 19 17:09:16.135636573 (263h43m10.566716695s) [noise=true , address=104.36.82.5:4853 , piece_id=K6TRSISY75NXYJNAAEALQ7W4RA26P5TGIVRMHX5SGOIF2GJ7X3GQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.135699574 (263h43m10.566779705s) [noise=true , address=104.36.82.60:4341 , piece_id=5NWVHAFJXPCBGPMRT42H4C3LPE3XKBDKPSX44D3DOWF6XXXJ5LFA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136028283 (263h43m10.567108404s) [noise=true , address=79.127.220.98:4432 , piece_id=4VA6KKCGA5GM55N3SD7FNUILD5XWYHE5VFRBFDIBAGVEK2VHQELA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136206604 (263h43m10.567286735s) [noise=true , address=104.36.82.22:4538 , piece_id=YUOOOPSZLQRNVU557APDO5FTOCY4FEYAHCBZQEE6YWMBPK5ZJYNA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136295745 (263h43m10.567375876s) [noise=true , address=77.90.31.11:4564 , piece_id=TX3C4VLEEOJOHBN7VDK4KIROG42PDDCB3SK26FZA7PM2455YZCMA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136640434 (263h43m10.567720565s) [noise=true , address=104.36.82.16:4315 , piece_id=P3SYICVJ5HSVY4MM62FDXQXJARZLIOQYNPBCT2SKYFPYZ3G3MSWQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136649702 (263h43m10.567729833s) [noise=true , address=104.36.82.44:4874 , piece_id=D3L2BM57MTGKIWKRSSEL7DRXUOK7LLKHL7M3Y3WAFCBILVDEV24Q, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136950888 (263h43m10.568031019s) [noise=true , address=104.36.82.35:4605 , piece_id=ORHH35ESP7KPGGDRD7ANECFHW7YTA2VFZTB5EMRCBQX5EWB7ZU4Q, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136961468 (263h43m10.568041599s) [noise=true , address=77.90.31.10:4535 , piece_id=N6LDVXERZAIR3EZ2TAMFQBM5Y565FMSP44I5KAWC32DTIOOM3FIQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137201027 (263h43m10.568281158s) [noise=true , address=77.90.31.12:4611 , piece_id=BANXKRHIMMOQFBSC4FW2E3SMNXCQERG2OTD7GXMM56T3X6JPNQGQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137260470 (263h43m10.568340601s) [noise=true , address=104.36.82.11:4134 , piece_id=BSGC4ADI6LMK3E2YEYPWFMYS3GC5VA42YM22VIJEGZ2JANYMLBYQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137463418 (263h43m10.568543549s) [noise=true , address=45.91.116.10:4110 , piece_id=KCZZOHPLSRNA45VXOMW7AKUYMRRFRUA52KSOBHNDRJPCB3RMV5HA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137553992 (263h43m10.568634123s) [noise=true , address=104.36.82.55:4224 , piece_id=PGZXUHGOONWDPIB4ZHAKLL4F2UWMZJPAL7BOTXIGSQWESLUPWGQA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137682447 (263h43m10.568762578s) [noise=true , address=45.91.116.12:4238 , piece_id=JAUKDCYIMCIOX4EFHQMCNCBLVZEIBKODPBL5QVBPOJIB554KLNRQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137883032 (263h43m10.568963163s) [noise=true , address=77.90.31.16:4743 , piece_id=YKPMLYPRMAZEBX7ORWEICH5AWKVS2J74D25W4ANIJJOUIBX5UBRA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137912418 (263h43m10.568992539s) [noise=true , address=104.36.82.23:4455 , piece_id=L2M5P5LRKVLVVDSYIOBV4SR3EAH2UO2JUZBGKZKIWBUIGFZPVEOA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.138136216 (263h43m10.569216347s) [noise=true , address=104.36.82.7:4930 , piece_id=EGTR52TZDAVTPPRCJLK4GOUXUWW2U7XJMQOV5P3JCKAKOK7GG5JQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.138374362 (263h43m10.569454493s) [noise=true , address=77.90.31.18:4839 , piece_id=BG553GCZSJEH5ORINSIQ7IUYU3UBQ3WO73ZAGRB7W45HOASJMBVQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:17.616571604 (263h43m12.047651765s) [upload=274176]: Next returned nil segment Aug 19 17:09:17.616591412 (263h43m12.047671543s) [upload=274176]: Waiting for error group managing segments... Aug 19 17:09:18.008239751 (263h43m12.439319882s) [noise=true , address=79.127.220.98:4432 , piece_id=4VA6KKCGA5GM55N3SD7FNUILD5XWYHE5VFRBFDIBAGVEK2VHQELA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.039911371 (263h43m12.470991502s) [noise=true , address=45.91.116.12:4238 , piece_id=JAUKDCYIMCIOX4EFHQMCNCBLVZEIBKODPBL5QVBPOJIB554KLNRQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.065695686 (263h43m12.496775817s) [noise=true , address=77.90.31.16:4743 , piece_id=YKPMLYPRMAZEBX7ORWEICH5AWKVS2J74D25W4ANIJJOUIBX5UBRA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.080085439 (263h43m12.51116557s) [noise=true , address=45.91.116.10:4110 , piece_id=KCZZOHPLSRNA45VXOMW7AKUYMRRFRUA52KSOBHNDRJPCB3RMV5HA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.131454157 (263h43m12.562534288s) [noise=true , address=104.36.82.11:4134 , piece_id=BSGC4ADI6LMK3E2YEYPWFMYS3GC5VA42YM22VIJEGZ2JANYMLBYQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.143557214 (263h43m12.574637345s) [noise=true , address=104.36.82.7:4930 , piece_id=EGTR52TZDAVTPPRCJLK4GOUXUWW2U7XJMQOV5P3JCKAKOK7GG5JQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.156135631 (263h43m12.587215762s) [noise=true , address=77.90.31.12:4611 , piece_id=BANXKRHIMMOQFBSC4FW2E3SMNXCQERG2OTD7GXMM56T3X6JPNQGQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.159345252 (263h43m12.590425373s) [noise=true , address=104.36.82.55:4224 , piece_id=PGZXUHGOONWDPIB4ZHAKLL4F2UWMZJPAL7BOTXIGSQWESLUPWGQA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.169106219 (263h43m12.60018635s) [noise=true , address=104.36.82.22:4538 , piece_id=YUOOOPSZLQRNVU557APDO5FTOCY4FEYAHCBZQEE6YWMBPK5ZJYNA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.188273593 (263h43m12.619353724s) [noise=true , address=104.36.82.35:4605 , piece_id=ORHH35ESP7KPGGDRD7ANECFHW7YTA2VFZTB5EMRCBQX5EWB7ZU4Q, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.204131846 (263h43m12.635211977s) [noise=true , address=104.36.82.16:4315 , piece_id=P3SYICVJ5HSVY4MM62FDXQXJARZLIOQYNPBCT2SKYFPYZ3G3MSWQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.217541965 (263h43m12.648622096s) [noise=true , address=104.36.82.60:4341 , piece_id=5NWVHAFJXPCBGPMRT42H4C3LPE3XKBDKPSX44D3DOWF6XXXJ5LFA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.222565827 (263h43m12.653645948s) [noise=true , address=104.36.82.23:4455 , piece_id=L2M5P5LRKVLVVDSYIOBV4SR3EAH2UO2JUZBGKZKIWBUIGFZPVEOA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.377619011 (263h43m12.808699132s) [noise=true , address=104.36.82.44:4874 , piece_id=D3L2BM57MTGKIWKRSSEL7DRXUOK7LLKHL7M3Y3WAFCBILVDEV24Q, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.377646604 (263h43m12.808726725s) [seg_pos={0 0}, upload=274176]: Segment reached optimal threshold of 14 pieces. Aug 19 17:09:18.377754100 (263h43m12.808834231s) [noise=true , address=104.36.82.5:4853 , piece_id=K6TRSISY75NXYJNAAEALQ7W4RA26P5TGIVRMHX5SGOIF2GJ7X3GQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377895570 (263h43m12.808975701s) [noise=true , address=77.90.31.18:4839 , piece_id=BG553GCZSJEH5ORINSIQ7IUYU3UBQ3WO73ZAGRB7W45HOASJMBVQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gmWwyaQ6zM6QdayBFpXdE9VSE1aaQcBHFcukJoASjjHWCDsNx): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377915529 (263h43m12.80899566s) [noise=true , address=77.90.31.11:4564 , piece_id=TX3C4VLEEOJOHBN7VDK4KIROG42PDDCB3SK26FZA7PM2455YZCMA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377974832 (263h43m12.809054963s) [noise=true , address=77.90.31.10:4535 , piece_id=N6LDVXERZAIR3EZ2TAMFQBM5Y565FMSP44I5KAWC32DTIOOM3FIQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377999008 (263h43m12.80907914s) [seg_pos={0 0}, upload=274176]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 17:09:18.378145599 (263h43m12.80922573s) [upload=274176]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 17:09:18.378150488 (263h43m12.809230619s) [upload=274176]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:09:18.983488776 (263h43m13.414568907s) [upload=274176]: Done uploading segments... Aug 19 17:09:18.983507222 (263h43m13.414587353s) [upload=274176]: Upload finished. err: <nil> Duration: 5.546294908s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [....X..........................................X............] duration: 2.111762615s FAIL (2025-08-19 17:13:29 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 17:04:06.343826144 (263h38m0.774906265s) [upload=274088]: Starting upload Aug 19 17:04:06.343875278 (263h38m0.774955409s) [upload=274088]: Done starting upload Aug 19 17:04:06.343971703 (263h38m0.775051834s) [upload=274088]: Uploading segments... Aug 19 17:04:06.344028321 (263h38m0.775108453s) [upload=274088]: Got next segment. Inline: true Aug 19 17:04:06.344033120 (263h38m0.775113252s) [upload=274088]: Waiting for error group managing segments... Aug 19 17:04:06.344068187 (263h38m0.775148318s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:04:06.344071964 (263h38m0.775152095s) [upload=274088]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:04:06.344074499 (263h38m0.77515462s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:04:06.653287654 (263h38m1.084367785s) [upload=274088]: Done uploading segments... Aug 19 17:04:06.653300519 (263h38m1.08438065s) [upload=274088]: Upload finished. err: <nil> Duration:* 2.11179194s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.77925046s FAIL (2025-08-19 17:13:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 263h18m33.75518017s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324344006 (263h18m33.755424127s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324633220 (263h18m33.755713351s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324851286 (263h18m33.755931417s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325068942 (263h18m33.756149063s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325283613 (263h18m33.756363744s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325499857 (263h18m33.756579998s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325759794 (263h18m33.756839915s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326059428 (263h18m33.757139549s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326277565 (263h18m33.757357696s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326470835 (263h18m33.757550966s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326673753 (263h18m33.757753884s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326867664 (263h18m33.757947795s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:40.851885389 (263h18m35.28296552s) [upload=273764]: Next returned nil segment Aug 19 16:44:40.851903334 (263h18m35.282983465s) [upload=273764]: Waiting for error group managing segments... Aug 19 16:44:41.244668372 (263h18m35.675748503s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.277280365 (263h18m35.708360496s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.328561665 (263h18m35.759641786s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.331747139 (263h18m35.76282727s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.357720320 (263h18m35.788800481s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.378190805 (263h18m35.809270936s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.486480333 (263h18m35.917560464s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:45:38.230570787 (263h19m32.661650918s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: expected piece hash; context deadline exceeded; EOF Aug 19 16:45:38.230979869 (263h19m32.66206s) [noise=true , address=77.90.31.18:4840 , piece_id=JWOETSFSZ3HELLZNLEASYNGW5OR64TMQVOFYYJAJVTHWCBHVTSLQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: read tcp 188.166.225.170:39946->77.90.31.18:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231017812 (263h19m32.662097943s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 188.166.225.170:44164->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231063489 (263h19m32.66214362s) [noise=true , address=45.91.116.14:4203 , piece_id=WRLBES3FI5EA5AQLZXJYGDROP3KBG3E2F22R4YD6A5KKI44PLV2Q, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 188.166.225.170:51738->45.91.116.14:4203: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231090952 (263h19m32.662171083s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: read tcp 188.166.225.170:57158->77.90.31.15:4707: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231110719 (263h19m32.66219085s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:50784->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231140165 (263h19m32.662220296s) [noise=true , address=77.90.31.16:4749 , piece_id=XCPXVBYLQH345FPIW4KDNLBFUCDDEW2H4B25WIXGOO5FVMAIVKPQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: read tcp 188.166.225.170:51668->77.90.31.16:4749: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231144814 (263h19m32.662224945s) [noise=true , address=77.90.31.19:4865 , piece_id=B3HUFIJZKVZJH4OBBO2DD7S3I7HX4LO2IQ74XJPO4DTW4QER35NA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1bguBghQonV5sTHaWteyUEnpp5KNzBKbvBpbvrUNFDCveHN77X, address:77.90.31.19:4865): protocol: read tcp 188.166.225.170:39398->77.90.31.19:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231177167 (263h19m32.662257298s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: read tcp 188.166.225.170:58526->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231184571 (263h19m32.662264702s) [noise=true , address=45.91.116.12:4236 , piece_id=LCRQS2GOSHOSLZBGQTGB3Q3PA4VN7KXNW66TDZ5O424TPJWTQE3A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH, address:45.91.116.12:4236): protocol: read tcp 188.166.225.170:57166->45.91.116.12:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231189770 (263h19m32.662269901s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 188.166.225.170:54242->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231215850 (263h19m32.662295981s) [seg_pos={0 0}, upload=273764]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 19 16:45:38.459100173 (263h19m32.890180304s) [upload=273764]: Done uploading segments... Aug 19 16:45:38.459128316 (263h19m32.890208447s) [upload=273764]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.779289055s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................X...X..................................] duration: 6.447792924s FAIL (2025-08-19 17:13:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.447819966s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.796812903s FAIL (2025-08-19 17:12:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 263h18m33.75518017s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324344006 (263h18m33.755424127s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324633220 (263h18m33.755713351s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324851286 (263h18m33.755931417s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325068942 (263h18m33.756149063s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325283613 (263h18m33.756363744s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325499857 (263h18m33.756579998s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325759794 (263h18m33.756839915s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326059428 (263h18m33.757139549s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326277565 (263h18m33.757357696s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326470835 (263h18m33.757550966s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326673753 (263h18m33.757753884s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326867664 (263h18m33.757947795s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:40.851885389 (263h18m35.28296552s) [upload=273764]: Next returned nil segment Aug 19 16:44:40.851903334 (263h18m35.282983465s) [upload=273764]: Waiting for error group managing segments... Aug 19 16:44:41.244668372 (263h18m35.675748503s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.277280365 (263h18m35.708360496s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.328561665 (263h18m35.759641786s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.331747139 (263h18m35.76282727s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.357720320 (263h18m35.788800481s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.378190805 (263h18m35.809270936s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.486480333 (263h18m35.917560464s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:45:38.230570787 (263h19m32.661650918s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: expected piece hash; context deadline exceeded; EOF Aug 19 16:45:38.230979869 (263h19m32.66206s) [noise=true , address=77.90.31.18:4840 , piece_id=JWOETSFSZ3HELLZNLEASYNGW5OR64TMQVOFYYJAJVTHWCBHVTSLQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: read tcp 188.166.225.170:39946->77.90.31.18:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231017812 (263h19m32.662097943s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 188.166.225.170:44164->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231063489 (263h19m32.66214362s) [noise=true , address=45.91.116.14:4203 , piece_id=WRLBES3FI5EA5AQLZXJYGDROP3KBG3E2F22R4YD6A5KKI44PLV2Q, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 188.166.225.170:51738->45.91.116.14:4203: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231090952 (263h19m32.662171083s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: read tcp 188.166.225.170:57158->77.90.31.15:4707: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231110719 (263h19m32.66219085s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:50784->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231140165 (263h19m32.662220296s) [noise=true , address=77.90.31.16:4749 , piece_id=XCPXVBYLQH345FPIW4KDNLBFUCDDEW2H4B25WIXGOO5FVMAIVKPQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: read tcp 188.166.225.170:51668->77.90.31.16:4749: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231144814 (263h19m32.662224945s) [noise=true , address=77.90.31.19:4865 , piece_id=B3HUFIJZKVZJH4OBBO2DD7S3I7HX4LO2IQ74XJPO4DTW4QER35NA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1bguBghQonV5sTHaWteyUEnpp5KNzBKbvBpbvrUNFDCveHN77X, address:77.90.31.19:4865): protocol: read tcp 188.166.225.170:39398->77.90.31.19:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231177167 (263h19m32.662257298s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: read tcp 188.166.225.170:58526->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231184571 (263h19m32.662264702s) [noise=true , address=45.91.116.12:4236 , piece_id=LCRQS2GOSHOSLZBGQTGB3Q3PA4VN7KXNW66TDZ5O424TPJWTQE3A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH, address:45.91.116.12:4236): protocol: read tcp 188.166.225.170:57166->45.91.116.12:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231189770 (263h19m32.662269901s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 188.166.225.170:54242->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231215850 (263h19m32.662295981s) [seg_pos={0 0}, upload=273764]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 19 16:45:38.459100173 (263h19m32.890180304s) [upload=273764]: Done uploading segments... Aug 19 16:45:38.459128316 (263h19m32.890208447s) [upload=273764]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.796843581s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X....X...............................] duration: 5.853670233s FAIL (2025-08-19 17:12:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.853692746s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X...X..............X......X.XX........................X....] duration: 6.063020149s FAIL (2025-08-19 17:12:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== Aug 19 17:09:15.684096389 (263h43m10.11517652s) [upload=274176]: Starting upload Aug 19 17:09:15.684143008 (263h43m10.115223139s) [upload=274176]: Done starting upload Aug 19 17:09:15.684256185 (263h43m10.115336306s) [upload=274176]: Uploading segments... Aug 19 17:09:15.684468862 (263h43m10.115548993s) [upload=274176]: Got next segment. Inline: false Aug 19 17:09:15.684504950 (263h43m10.115585082s) [upload=274176]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:09:15.684509770 (263h43m10.115589901s) [upload=274176]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:09:16.135567491 (263h43m10.566647622s) [seg_pos={0 0}, upload=274176]: Begin upload segment... Aug 19 17:09:16.135613589 (263h43m10.56669372s) [seg_pos={0 0}, upload=274176]: Done begin upload segment. Aug 19 17:09:16.135636573 (263h43m10.566716695s) [noise=true , address=104.36.82.5:4853 , piece_id=K6TRSISY75NXYJNAAEALQ7W4RA26P5TGIVRMHX5SGOIF2GJ7X3GQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.135699574 (263h43m10.566779705s) [noise=true , address=104.36.82.60:4341 , piece_id=5NWVHAFJXPCBGPMRT42H4C3LPE3XKBDKPSX44D3DOWF6XXXJ5LFA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136028283 (263h43m10.567108404s) [noise=true , address=79.127.220.98:4432 , piece_id=4VA6KKCGA5GM55N3SD7FNUILD5XWYHE5VFRBFDIBAGVEK2VHQELA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136206604 (263h43m10.567286735s) [noise=true , address=104.36.82.22:4538 , piece_id=YUOOOPSZLQRNVU557APDO5FTOCY4FEYAHCBZQEE6YWMBPK5ZJYNA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136295745 (263h43m10.567375876s) [noise=true , address=77.90.31.11:4564 , piece_id=TX3C4VLEEOJOHBN7VDK4KIROG42PDDCB3SK26FZA7PM2455YZCMA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136640434 (263h43m10.567720565s) [noise=true , address=104.36.82.16:4315 , piece_id=P3SYICVJ5HSVY4MM62FDXQXJARZLIOQYNPBCT2SKYFPYZ3G3MSWQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136649702 (263h43m10.567729833s) [noise=true , address=104.36.82.44:4874 , piece_id=D3L2BM57MTGKIWKRSSEL7DRXUOK7LLKHL7M3Y3WAFCBILVDEV24Q, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136950888 (263h43m10.568031019s) [noise=true , address=104.36.82.35:4605 , piece_id=ORHH35ESP7KPGGDRD7ANECFHW7YTA2VFZTB5EMRCBQX5EWB7ZU4Q, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136961468 (263h43m10.568041599s) [noise=true , address=77.90.31.10:4535 , piece_id=N6LDVXERZAIR3EZ2TAMFQBM5Y565FMSP44I5KAWC32DTIOOM3FIQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137201027 (263h43m10.568281158s) [noise=true , address=77.90.31.12:4611 , piece_id=BANXKRHIMMOQFBSC4FW2E3SMNXCQERG2OTD7GXMM56T3X6JPNQGQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137260470 (263h43m10.568340601s) [noise=true , address=104.36.82.11:4134 , piece_id=BSGC4ADI6LMK3E2YEYPWFMYS3GC5VA42YM22VIJEGZ2JANYMLBYQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137463418 (263h43m10.568543549s) [noise=true , address=45.91.116.10:4110 , piece_id=KCZZOHPLSRNA45VXOMW7AKUYMRRFRUA52KSOBHNDRJPCB3RMV5HA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137553992 (263h43m10.568634123s) [noise=true , address=104.36.82.55:4224 , piece_id=PGZXUHGOONWDPIB4ZHAKLL4F2UWMZJPAL7BOTXIGSQWESLUPWGQA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137682447 (263h43m10.568762578s) [noise=true , address=45.91.116.12:4238 , piece_id=JAUKDCYIMCIOX4EFHQMCNCBLVZEIBKODPBL5QVBPOJIB554KLNRQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137883032 (263h43m10.568963163s) [noise=true , address=77.90.31.16:4743 , piece_id=YKPMLYPRMAZEBX7ORWEICH5AWKVS2J74D25W4ANIJJOUIBX5UBRA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137912418 (263h43m10.568992539s) [noise=true , address=104.36.82.23:4455 , piece_id=L2M5P5LRKVLVVDSYIOBV4SR3EAH2UO2JUZBGKZKIWBUIGFZPVEOA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.138136216 (263h43m10.569216347s) [noise=true , address=104.36.82.7:4930 , piece_id=EGTR52TZDAVTPPRCJLK4GOUXUWW2U7XJMQOV5P3JCKAKOK7GG5JQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.138374362 (263h43m10.569454493s) [noise=true , address=77.90.31.18:4839 , piece_id=BG553GCZSJEH5ORINSIQ7IUYU3UBQ3WO73ZAGRB7W45HOASJMBVQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:17.616571604 (263h43m12.047651765s) [upload=274176]: Next returned nil segment Aug 19 17:09:17.616591412 (263h43m12.047671543s) [upload=274176]: Waiting for error group managing segments... Aug 19 17:09:18.008239751 (263h43m12.439319882s) [noise=true , address=79.127.220.98:4432 , piece_id=4VA6KKCGA5GM55N3SD7FNUILD5XWYHE5VFRBFDIBAGVEK2VHQELA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.039911371 (263h43m12.470991502s) [noise=true , address=45.91.116.12:4238 , piece_id=JAUKDCYIMCIOX4EFHQMCNCBLVZEIBKODPBL5QVBPOJIB554KLNRQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.065695686 (263h43m12.496775817s) [noise=true , address=77.90.31.16:4743 , piece_id=YKPMLYPRMAZEBX7ORWEICH5AWKVS2J74D25W4ANIJJOUIBX5UBRA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.080085439 (263h43m12.51116557s) [noise=true , address=45.91.116.10:4110 , piece_id=KCZZOHPLSRNA45VXOMW7AKUYMRRFRUA52KSOBHNDRJPCB3RMV5HA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.131454157 (263h43m12.562534288s) [noise=true , address=104.36.82.11:4134 , piece_id=BSGC4ADI6LMK3E2YEYPWFMYS3GC5VA42YM22VIJEGZ2JANYMLBYQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.143557214 (263h43m12.574637345s) [noise=true , address=104.36.82.7:4930 , piece_id=EGTR52TZDAVTPPRCJLK4GOUXUWW2U7XJMQOV5P3JCKAKOK7GG5JQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.156135631 (263h43m12.587215762s) [noise=true , address=77.90.31.12:4611 , piece_id=BANXKRHIMMOQFBSC4FW2E3SMNXCQERG2OTD7GXMM56T3X6JPNQGQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.159345252 (263h43m12.590425373s) [noise=true , address=104.36.82.55:4224 , piece_id=PGZXUHGOONWDPIB4ZHAKLL4F2UWMZJPAL7BOTXIGSQWESLUPWGQA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.169106219 (263h43m12.60018635s) [noise=true , address=104.36.82.22:4538 , piece_id=YUOOOPSZLQRNVU557APDO5FTOCY4FEYAHCBZQEE6YWMBPK5ZJYNA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.188273593 (263h43m12.619353724s) [noise=true , address=104.36.82.35:4605 , piece_id=ORHH35ESP7KPGGDRD7ANECFHW7YTA2VFZTB5EMRCBQX5EWB7ZU4Q, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.204131846 (263h43m12.635211977s) [noise=true , address=104.36.82.16:4315 , piece_id=P3SYICVJ5HSVY4MM62FDXQXJARZLIOQYNPBCT2SKYFPYZ3G3MSWQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.217541965 (263h43m12.648622096s) [noise=true , address=104.36.82.60:4341 , piece_id=5NWVHAFJXPCBGPMRT42H4C3LPE3XKBDKPSX44D3DOWF6XXXJ5LFA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.222565827 (263h43m12.653645948s) [noise=true , address=104.36.82.23:4455 , piece_id=L2M5P5LRKVLVVDSYIOBV4SR3EAH2UO2JUZBGKZKIWBUIGFZPVEOA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.377619011 (263h43m12.808699132s) [noise=true , address=104.36.82.44:4874 , piece_id=D3L2BM57MTGKIWKRSSEL7DRXUOK7LLKHL7M3Y3WAFCBILVDEV24Q, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.377646604 (263h43m12.808726725s) [seg_pos={0 0}, upload=274176]: Segment reached optimal threshold of 14 pieces. Aug 19 17:09:18.377754100 (263h43m12.808834231s) [noise=true , address=104.36.82.5:4853 , piece_id=K6TRSISY75NXYJNAAEALQ7W4RA26P5TGIVRMHX5SGOIF2GJ7X3GQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377895570 (263h43m12.808975701s) [noise=true , address=77.90.31.18:4839 , piece_id=BG553GCZSJEH5ORINSIQ7IUYU3UBQ3WO73ZAGRB7W45HOASJMBVQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gmWwyaQ6zM6QdayBFpXdE9VSE1aaQcBHFcukJoASjjHWCDsNx): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377915529 (263h43m12.80899566s) [noise=true , address=77.90.31.11:4564 , piece_id=TX3C4VLEEOJOHBN7VDK4KIROG42PDDCB3SK26FZA7PM2455YZCMA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377974832 (263h43m12.809054963s) [noise=true , address=77.90.31.10:4535 , piece_id=N6LDVXERZAIR3EZ2TAMFQBM5Y565FMSP44I5KAWC32DTIOOM3FIQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377999008 (263h43m12.80907914s) [seg_pos={0 0}, upload=274176]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 17:09:18.378145599 (263h43m12.80922573s) [upload=274176]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 17:09:18.378150488 (263h43m12.809230619s) [upload=274176]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:09:18.983488776 (263h43m13.414568907s) [upload=274176]: Done uploading segments... Aug 19 17:09:18.983507222 (263h43m13.414587353s) [upload=274176]: Upload finished. err: <nil> Duration: 6.063047851s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X....XX...............................] duration: 5.897478667s FAIL (2025-08-19 17:12:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 19 16:49:12.801154313 (263h23m7.232234445s) [upload=273839]: Starting upload Aug 19 16:49:12.801215942 (263h23m7.232296073s) [upload=273839]: Done starting upload Aug 19 16:49:12.801330351 (263h23m7.232410482s) [upload=273839]: Uploading segments... Aug 19 16:49:12.801425372 (263h23m7.232505503s) [upload=273839]: Got next segment. Inline: false Aug 19 16:49:12.801455880 (263h23m7.232536012s) [upload=273839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:49:12.801461982 (263h23m7.232542113s) [upload=273839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:49:13.036708228 (263h23m7.467788359s) [seg_pos={0 0}, upload=273839]: Begin upload segment... Aug 19 16:49:13.036749197 (263h23m7.467829328s) [seg_pos={0 0}, upload=273839]: Done begin upload segment. Aug 19 16:49:13.036771971 (263h23m7.467852092s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.036780256 (263h23m7.467860437s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037135917 (263h23m7.468216048s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037326592 (263h23m7.468406723s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037403259 (263h23m7.46848339s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037667814 (263h23m7.468747945s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037761764 (263h23m7.468841896s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037943331 (263h23m7.469023462s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038070565 (263h23m7.469150696s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038168622 (263h23m7.469248753s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038300104 (263h23m7.469380235s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038396789 (263h23m7.46947692s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038622241 (263h23m7.469702362s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038679350 (263h23m7.469759481s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038854735 (263h23m7.469934856s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.039048045 (263h23m7.470128166s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:14.749064365 (263h23m9.180144496s) [upload=273839]: Next returned nil segment Aug 19 16:49:14.749117217 (263h23m9.180197348s) [upload=273839]: Waiting for error group managing segments... Aug 19 16:49:15.090317863 (263h23m9.521397994s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.132109007 (263h23m9.563189138s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.222090156 (263h23m9.653170287s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.328041780 (263h23m9.759121911s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.353998781 (263h23m9.785078912s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:36.709543777 (263h23m31.140623898s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:37.041062020 (263h23m31.472142141s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.183707791 (263h23m32.614787922s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.694075084 (263h23m33.125155215s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.716788163 (263h23m34.147868294s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.851890717 (263h23m34.282970848s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:40.293306092 (263h23m34.724386243s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:42.156929229 (263h23m36.58800936s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157075889 (263h23m36.58815601s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157126075 (263h23m36.588206206s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection Aug 19 16:49:42.157170951 (263h23m36.588251082s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: read tcp 188.166.225.170:34242->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:42.157215747 (263h23m36.588295878s) [seg_pos={0 0}, upload=273839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 19 16:49:42.385478364 (263h23m36.816558495s) [upload=273839]: Done uploading segments... Aug 19 16:49:42.385503091 (263h23m36.816583212s) [upload=273839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.897511008s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....X..........................................X...........] duration: 1.998738161s FAIL (2025-08-19 17:12:20 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 17:04:06.343826144 (263h38m0.774906265s) [upload=274088]: Starting upload Aug 19 17:04:06.343875278 (263h38m0.774955409s) [upload=274088]: Done starting upload Aug 19 17:04:06.343971703 (263h38m0.775051834s) [upload=274088]: Uploading segments... Aug 19 17:04:06.344028321 (263h38m0.775108453s) [upload=274088]: Got next segment. Inline: true Aug 19 17:04:06.344033120 (263h38m0.775113252s) [upload=274088]: Waiting for error group managing segments... Aug 19 17:04:06.344068187 (263h38m0.775148318s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:04:06.344071964 (263h38m0.775152095s) [upload=274088]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:04:06.344074499 (263h38m0.77515462s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:04:06.653287654 (263h38m1.084367785s) [upload=274088]: Done uploading segments... Aug 19 17:04:06.653300519 (263h38m1.08438065s) [upload=274088]: Upload finished. err: <nil> Duration:* 1.998762107s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................X...X.................................] duration: 6.299057304s FAIL (2025-08-19 17:12:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.299078815s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.571802071s FAIL (2025-08-19 17:12:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 263h18m33.75518017s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324344006 (263h18m33.755424127s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324633220 (263h18m33.755713351s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324851286 (263h18m33.755931417s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325068942 (263h18m33.756149063s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325283613 (263h18m33.756363744s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325499857 (263h18m33.756579998s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325759794 (263h18m33.756839915s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326059428 (263h18m33.757139549s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326277565 (263h18m33.757357696s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326470835 (263h18m33.757550966s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326673753 (263h18m33.757753884s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326867664 (263h18m33.757947795s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:40.851885389 (263h18m35.28296552s) [upload=273764]: Next returned nil segment Aug 19 16:44:40.851903334 (263h18m35.282983465s) [upload=273764]: Waiting for error group managing segments... Aug 19 16:44:41.244668372 (263h18m35.675748503s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.277280365 (263h18m35.708360496s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.328561665 (263h18m35.759641786s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.331747139 (263h18m35.76282727s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.357720320 (263h18m35.788800481s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.378190805 (263h18m35.809270936s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.486480333 (263h18m35.917560464s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:45:38.230570787 (263h19m32.661650918s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: expected piece hash; context deadline exceeded; EOF Aug 19 16:45:38.230979869 (263h19m32.66206s) [noise=true , address=77.90.31.18:4840 , piece_id=JWOETSFSZ3HELLZNLEASYNGW5OR64TMQVOFYYJAJVTHWCBHVTSLQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: read tcp 188.166.225.170:39946->77.90.31.18:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231017812 (263h19m32.662097943s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 188.166.225.170:44164->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231063489 (263h19m32.66214362s) [noise=true , address=45.91.116.14:4203 , piece_id=WRLBES3FI5EA5AQLZXJYGDROP3KBG3E2F22R4YD6A5KKI44PLV2Q, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 188.166.225.170:51738->45.91.116.14:4203: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231090952 (263h19m32.662171083s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: read tcp 188.166.225.170:57158->77.90.31.15:4707: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231110719 (263h19m32.66219085s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:50784->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231140165 (263h19m32.662220296s) [noise=true , address=77.90.31.16:4749 , piece_id=XCPXVBYLQH345FPIW4KDNLBFUCDDEW2H4B25WIXGOO5FVMAIVKPQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: read tcp 188.166.225.170:51668->77.90.31.16:4749: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231144814 (263h19m32.662224945s) [noise=true , address=77.90.31.19:4865 , piece_id=B3HUFIJZKVZJH4OBBO2DD7S3I7HX4LO2IQ74XJPO4DTW4QER35NA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1bguBghQonV5sTHaWteyUEnpp5KNzBKbvBpbvrUNFDCveHN77X, address:77.90.31.19:4865): protocol: read tcp 188.166.225.170:39398->77.90.31.19:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231177167 (263h19m32.662257298s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: read tcp 188.166.225.170:58526->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231184571 (263h19m32.662264702s) [noise=true , address=45.91.116.12:4236 , piece_id=LCRQS2GOSHOSLZBGQTGB3Q3PA4VN7KXNW66TDZ5O424TPJWTQE3A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH, address:45.91.116.12:4236): protocol: read tcp 188.166.225.170:57166->45.91.116.12:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231189770 (263h19m32.662269901s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 188.166.225.170:54242->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231215850 (263h19m32.662295981s) [seg_pos={0 0}, upload=273764]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 19 16:45:38.459100173 (263h19m32.890180304s) [upload=273764]: Done uploading segments... Aug 19 16:45:38.459128316 (263h19m32.890208447s) [upload=273764]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.571825526s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X....X..............................] duration: 5.864693872s FAIL (2025-08-19 17:12:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.86471877s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X...X..............X......X.XX........................X...] duration: 6.003479339s FAIL (2025-08-19 17:11:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== Aug 19 17:09:15.684096389 (263h43m10.11517652s) [upload=274176]: Starting upload Aug 19 17:09:15.684143008 (263h43m10.115223139s) [upload=274176]: Done starting upload Aug 19 17:09:15.684256185 (263h43m10.115336306s) [upload=274176]: Uploading segments... Aug 19 17:09:15.684468862 (263h43m10.115548993s) [upload=274176]: Got next segment. Inline: false Aug 19 17:09:15.684504950 (263h43m10.115585082s) [upload=274176]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:09:15.684509770 (263h43m10.115589901s) [upload=274176]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:09:16.135567491 (263h43m10.566647622s) [seg_pos={0 0}, upload=274176]: Begin upload segment... Aug 19 17:09:16.135613589 (263h43m10.56669372s) [seg_pos={0 0}, upload=274176]: Done begin upload segment. Aug 19 17:09:16.135636573 (263h43m10.566716695s) [noise=true , address=104.36.82.5:4853 , piece_id=K6TRSISY75NXYJNAAEALQ7W4RA26P5TGIVRMHX5SGOIF2GJ7X3GQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.135699574 (263h43m10.566779705s) [noise=true , address=104.36.82.60:4341 , piece_id=5NWVHAFJXPCBGPMRT42H4C3LPE3XKBDKPSX44D3DOWF6XXXJ5LFA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136028283 (263h43m10.567108404s) [noise=true , address=79.127.220.98:4432 , piece_id=4VA6KKCGA5GM55N3SD7FNUILD5XWYHE5VFRBFDIBAGVEK2VHQELA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136206604 (263h43m10.567286735s) [noise=true , address=104.36.82.22:4538 , piece_id=YUOOOPSZLQRNVU557APDO5FTOCY4FEYAHCBZQEE6YWMBPK5ZJYNA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136295745 (263h43m10.567375876s) [noise=true , address=77.90.31.11:4564 , piece_id=TX3C4VLEEOJOHBN7VDK4KIROG42PDDCB3SK26FZA7PM2455YZCMA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136640434 (263h43m10.567720565s) [noise=true , address=104.36.82.16:4315 , piece_id=P3SYICVJ5HSVY4MM62FDXQXJARZLIOQYNPBCT2SKYFPYZ3G3MSWQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136649702 (263h43m10.567729833s) [noise=true , address=104.36.82.44:4874 , piece_id=D3L2BM57MTGKIWKRSSEL7DRXUOK7LLKHL7M3Y3WAFCBILVDEV24Q, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136950888 (263h43m10.568031019s) [noise=true , address=104.36.82.35:4605 , piece_id=ORHH35ESP7KPGGDRD7ANECFHW7YTA2VFZTB5EMRCBQX5EWB7ZU4Q, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136961468 (263h43m10.568041599s) [noise=true , address=77.90.31.10:4535 , piece_id=N6LDVXERZAIR3EZ2TAMFQBM5Y565FMSP44I5KAWC32DTIOOM3FIQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137201027 (263h43m10.568281158s) [noise=true , address=77.90.31.12:4611 , piece_id=BANXKRHIMMOQFBSC4FW2E3SMNXCQERG2OTD7GXMM56T3X6JPNQGQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137260470 (263h43m10.568340601s) [noise=true , address=104.36.82.11:4134 , piece_id=BSGC4ADI6LMK3E2YEYPWFMYS3GC5VA42YM22VIJEGZ2JANYMLBYQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137463418 (263h43m10.568543549s) [noise=true , address=45.91.116.10:4110 , piece_id=KCZZOHPLSRNA45VXOMW7AKUYMRRFRUA52KSOBHNDRJPCB3RMV5HA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137553992 (263h43m10.568634123s) [noise=true , address=104.36.82.55:4224 , piece_id=PGZXUHGOONWDPIB4ZHAKLL4F2UWMZJPAL7BOTXIGSQWESLUPWGQA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137682447 (263h43m10.568762578s) [noise=true , address=45.91.116.12:4238 , piece_id=JAUKDCYIMCIOX4EFHQMCNCBLVZEIBKODPBL5QVBPOJIB554KLNRQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137883032 (263h43m10.568963163s) [noise=true , address=77.90.31.16:4743 , piece_id=YKPMLYPRMAZEBX7ORWEICH5AWKVS2J74D25W4ANIJJOUIBX5UBRA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137912418 (263h43m10.568992539s) [noise=true , address=104.36.82.23:4455 , piece_id=L2M5P5LRKVLVVDSYIOBV4SR3EAH2UO2JUZBGKZKIWBUIGFZPVEOA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.138136216 (263h43m10.569216347s) [noise=true , address=104.36.82.7:4930 , piece_id=EGTR52TZDAVTPPRCJLK4GOUXUWW2U7XJMQOV5P3JCKAKOK7GG5JQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.138374362 (263h43m10.569454493s) [noise=true , address=77.90.31.18:4839 , piece_id=BG553GCZSJEH5ORINSIQ7IUYU3UBQ3WO73ZAGRB7W45HOASJMBVQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:17.616571604 (263h43m12.047651765s) [upload=274176]: Next returned nil segment Aug 19 17:09:17.616591412 (263h43m12.047671543s) [upload=274176]: Waiting for error group managing segments... Aug 19 17:09:18.008239751 (263h43m12.439319882s) [noise=true , address=79.127.220.98:4432 , piece_id=4VA6KKCGA5GM55N3SD7FNUILD5XWYHE5VFRBFDIBAGVEK2VHQELA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.039911371 (263h43m12.470991502s) [noise=true , address=45.91.116.12:4238 , piece_id=JAUKDCYIMCIOX4EFHQMCNCBLVZEIBKODPBL5QVBPOJIB554KLNRQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.065695686 (263h43m12.496775817s) [noise=true , address=77.90.31.16:4743 , piece_id=YKPMLYPRMAZEBX7ORWEICH5AWKVS2J74D25W4ANIJJOUIBX5UBRA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.080085439 (263h43m12.51116557s) [noise=true , address=45.91.116.10:4110 , piece_id=KCZZOHPLSRNA45VXOMW7AKUYMRRFRUA52KSOBHNDRJPCB3RMV5HA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.131454157 (263h43m12.562534288s) [noise=true , address=104.36.82.11:4134 , piece_id=BSGC4ADI6LMK3E2YEYPWFMYS3GC5VA42YM22VIJEGZ2JANYMLBYQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.143557214 (263h43m12.574637345s) [noise=true , address=104.36.82.7:4930 , piece_id=EGTR52TZDAVTPPRCJLK4GOUXUWW2U7XJMQOV5P3JCKAKOK7GG5JQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.156135631 (263h43m12.587215762s) [noise=true , address=77.90.31.12:4611 , piece_id=BANXKRHIMMOQFBSC4FW2E3SMNXCQERG2OTD7GXMM56T3X6JPNQGQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.159345252 (263h43m12.590425373s) [noise=true , address=104.36.82.55:4224 , piece_id=PGZXUHGOONWDPIB4ZHAKLL4F2UWMZJPAL7BOTXIGSQWESLUPWGQA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.169106219 (263h43m12.60018635s) [noise=true , address=104.36.82.22:4538 , piece_id=YUOOOPSZLQRNVU557APDO5FTOCY4FEYAHCBZQEE6YWMBPK5ZJYNA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.188273593 (263h43m12.619353724s) [noise=true , address=104.36.82.35:4605 , piece_id=ORHH35ESP7KPGGDRD7ANECFHW7YTA2VFZTB5EMRCBQX5EWB7ZU4Q, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.204131846 (263h43m12.635211977s) [noise=true , address=104.36.82.16:4315 , piece_id=P3SYICVJ5HSVY4MM62FDXQXJARZLIOQYNPBCT2SKYFPYZ3G3MSWQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.217541965 (263h43m12.648622096s) [noise=true , address=104.36.82.60:4341 , piece_id=5NWVHAFJXPCBGPMRT42H4C3LPE3XKBDKPSX44D3DOWF6XXXJ5LFA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.222565827 (263h43m12.653645948s) [noise=true , address=104.36.82.23:4455 , piece_id=L2M5P5LRKVLVVDSYIOBV4SR3EAH2UO2JUZBGKZKIWBUIGFZPVEOA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.377619011 (263h43m12.808699132s) [noise=true , address=104.36.82.44:4874 , piece_id=D3L2BM57MTGKIWKRSSEL7DRXUOK7LLKHL7M3Y3WAFCBILVDEV24Q, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.377646604 (263h43m12.808726725s) [seg_pos={0 0}, upload=274176]: Segment reached optimal threshold of 14 pieces. Aug 19 17:09:18.377754100 (263h43m12.808834231s) [noise=true , address=104.36.82.5:4853 , piece_id=K6TRSISY75NXYJNAAEALQ7W4RA26P5TGIVRMHX5SGOIF2GJ7X3GQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377895570 (263h43m12.808975701s) [noise=true , address=77.90.31.18:4839 , piece_id=BG553GCZSJEH5ORINSIQ7IUYU3UBQ3WO73ZAGRB7W45HOASJMBVQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gmWwyaQ6zM6QdayBFpXdE9VSE1aaQcBHFcukJoASjjHWCDsNx): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377915529 (263h43m12.80899566s) [noise=true , address=77.90.31.11:4564 , piece_id=TX3C4VLEEOJOHBN7VDK4KIROG42PDDCB3SK26FZA7PM2455YZCMA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377974832 (263h43m12.809054963s) [noise=true , address=77.90.31.10:4535 , piece_id=N6LDVXERZAIR3EZ2TAMFQBM5Y565FMSP44I5KAWC32DTIOOM3FIQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377999008 (263h43m12.80907914s) [seg_pos={0 0}, upload=274176]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 17:09:18.378145599 (263h43m12.80922573s) [upload=274176]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 17:09:18.378150488 (263h43m12.809230619s) [upload=274176]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:09:18.983488776 (263h43m13.414568907s) [upload=274176]: Done uploading segments... Aug 19 17:09:18.983507222 (263h43m13.414587353s) [upload=274176]: Upload finished. err: <nil> Duration: 6.003558309s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.617678434s FAIL (2025-08-19 17:11:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 263h18m33.75518017s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324344006 (263h18m33.755424127s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324633220 (263h18m33.755713351s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324851286 (263h18m33.755931417s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325068942 (263h18m33.756149063s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325283613 (263h18m33.756363744s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325499857 (263h18m33.756579998s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325759794 (263h18m33.756839915s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326059428 (263h18m33.757139549s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326277565 (263h18m33.757357696s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326470835 (263h18m33.757550966s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326673753 (263h18m33.757753884s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326867664 (263h18m33.757947795s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:40.851885389 (263h18m35.28296552s) [upload=273764]: Next returned nil segment Aug 19 16:44:40.851903334 (263h18m35.282983465s) [upload=273764]: Waiting for error group managing segments... Aug 19 16:44:41.244668372 (263h18m35.675748503s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.277280365 (263h18m35.708360496s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.328561665 (263h18m35.759641786s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.331747139 (263h18m35.76282727s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.357720320 (263h18m35.788800481s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.378190805 (263h18m35.809270936s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.486480333 (263h18m35.917560464s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:45:38.230570787 (263h19m32.661650918s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: expected piece hash; context deadline exceeded; EOF Aug 19 16:45:38.230979869 (263h19m32.66206s) [noise=true , address=77.90.31.18:4840 , piece_id=JWOETSFSZ3HELLZNLEASYNGW5OR64TMQVOFYYJAJVTHWCBHVTSLQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: read tcp 188.166.225.170:39946->77.90.31.18:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231017812 (263h19m32.662097943s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 188.166.225.170:44164->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231063489 (263h19m32.66214362s) [noise=true , address=45.91.116.14:4203 , piece_id=WRLBES3FI5EA5AQLZXJYGDROP3KBG3E2F22R4YD6A5KKI44PLV2Q, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 188.166.225.170:51738->45.91.116.14:4203: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231090952 (263h19m32.662171083s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: read tcp 188.166.225.170:57158->77.90.31.15:4707: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231110719 (263h19m32.66219085s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:50784->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231140165 (263h19m32.662220296s) [noise=true , address=77.90.31.16:4749 , piece_id=XCPXVBYLQH345FPIW4KDNLBFUCDDEW2H4B25WIXGOO5FVMAIVKPQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: read tcp 188.166.225.170:51668->77.90.31.16:4749: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231144814 (263h19m32.662224945s) [noise=true , address=77.90.31.19:4865 , piece_id=B3HUFIJZKVZJH4OBBO2DD7S3I7HX4LO2IQ74XJPO4DTW4QER35NA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1bguBghQonV5sTHaWteyUEnpp5KNzBKbvBpbvrUNFDCveHN77X, address:77.90.31.19:4865): protocol: read tcp 188.166.225.170:39398->77.90.31.19:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231177167 (263h19m32.662257298s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: read tcp 188.166.225.170:58526->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231184571 (263h19m32.662264702s) [noise=true , address=45.91.116.12:4236 , piece_id=LCRQS2GOSHOSLZBGQTGB3Q3PA4VN7KXNW66TDZ5O424TPJWTQE3A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH, address:45.91.116.12:4236): protocol: read tcp 188.166.225.170:57166->45.91.116.12:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231189770 (263h19m32.662269901s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 188.166.225.170:54242->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231215850 (263h19m32.662295981s) [seg_pos={0 0}, upload=273764]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 19 16:45:38.459100173 (263h19m32.890180304s) [upload=273764]: Done uploading segments... Aug 19 16:45:38.459128316 (263h19m32.890208447s) [upload=273764]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.61770742s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X....XX..............................] duration: 5.811861248s FAIL (2025-08-19 17:11:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 19 16:49:12.801154313 (263h23m7.232234445s) [upload=273839]: Starting upload Aug 19 16:49:12.801215942 (263h23m7.232296073s) [upload=273839]: Done starting upload Aug 19 16:49:12.801330351 (263h23m7.232410482s) [upload=273839]: Uploading segments... Aug 19 16:49:12.801425372 (263h23m7.232505503s) [upload=273839]: Got next segment. Inline: false Aug 19 16:49:12.801455880 (263h23m7.232536012s) [upload=273839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:49:12.801461982 (263h23m7.232542113s) [upload=273839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:49:13.036708228 (263h23m7.467788359s) [seg_pos={0 0}, upload=273839]: Begin upload segment... Aug 19 16:49:13.036749197 (263h23m7.467829328s) [seg_pos={0 0}, upload=273839]: Done begin upload segment. Aug 19 16:49:13.036771971 (263h23m7.467852092s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.036780256 (263h23m7.467860437s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037135917 (263h23m7.468216048s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037326592 (263h23m7.468406723s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037403259 (263h23m7.46848339s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037667814 (263h23m7.468747945s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037761764 (263h23m7.468841896s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037943331 (263h23m7.469023462s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038070565 (263h23m7.469150696s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038168622 (263h23m7.469248753s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038300104 (263h23m7.469380235s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038396789 (263h23m7.46947692s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038622241 (263h23m7.469702362s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038679350 (263h23m7.469759481s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038854735 (263h23m7.469934856s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.039048045 (263h23m7.470128166s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:14.749064365 (263h23m9.180144496s) [upload=273839]: Next returned nil segment Aug 19 16:49:14.749117217 (263h23m9.180197348s) [upload=273839]: Waiting for error group managing segments... Aug 19 16:49:15.090317863 (263h23m9.521397994s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.132109007 (263h23m9.563189138s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.222090156 (263h23m9.653170287s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.328041780 (263h23m9.759121911s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.353998781 (263h23m9.785078912s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:36.709543777 (263h23m31.140623898s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:37.041062020 (263h23m31.472142141s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.183707791 (263h23m32.614787922s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.694075084 (263h23m33.125155215s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.716788163 (263h23m34.147868294s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.851890717 (263h23m34.282970848s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:40.293306092 (263h23m34.724386243s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:42.156929229 (263h23m36.58800936s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157075889 (263h23m36.58815601s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157126075 (263h23m36.588206206s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection Aug 19 16:49:42.157170951 (263h23m36.588251082s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: read tcp 188.166.225.170:34242->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:42.157215747 (263h23m36.588295878s) [seg_pos={0 0}, upload=273839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 19 16:49:42.385478364 (263h23m36.816558495s) [upload=273839]: Done uploading segments... Aug 19 16:49:42.385503091 (263h23m36.816583212s) [upload=273839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.811886066s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................X...X................................] duration: 6.46765231s FAIL (2025-08-19 17:11:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.4676732s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [......X..........................................X..........] duration: 2.053676828s FAIL (2025-08-19 17:11:14 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 17:04:06.343826144 (263h38m0.774906265s) [upload=274088]: Starting upload Aug 19 17:04:06.343875278 (263h38m0.774955409s) [upload=274088]: Done starting upload Aug 19 17:04:06.343971703 (263h38m0.775051834s) [upload=274088]: Uploading segments... Aug 19 17:04:06.344028321 (263h38m0.775108453s) [upload=274088]: Got next segment. Inline: true Aug 19 17:04:06.344033120 (263h38m0.775113252s) [upload=274088]: Waiting for error group managing segments... Aug 19 17:04:06.344068187 (263h38m0.775148318s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:04:06.344071964 (263h38m0.775152095s) [upload=274088]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:04:06.344074499 (263h38m0.77515462s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:04:06.653287654 (263h38m1.084367785s) [upload=274088]: Done uploading segments... Aug 19 17:04:06.653300519 (263h38m1.08438065s) [upload=274088]: Upload finished. err: <nil> Duration:* 2.05371924s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X....X.............................] duration: 6.175927732s FAIL (2025-08-19 17:11:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.175955714s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.812925213s FAIL (2025-08-19 17:11:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 263h18m33.75518017s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324344006 (263h18m33.755424127s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324633220 (263h18m33.755713351s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324851286 (263h18m33.755931417s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325068942 (263h18m33.756149063s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325283613 (263h18m33.756363744s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325499857 (263h18m33.756579998s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325759794 (263h18m33.756839915s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326059428 (263h18m33.757139549s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326277565 (263h18m33.757357696s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326470835 (263h18m33.757550966s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326673753 (263h18m33.757753884s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326867664 (263h18m33.757947795s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:40.851885389 (263h18m35.28296552s) [upload=273764]: Next returned nil segment Aug 19 16:44:40.851903334 (263h18m35.282983465s) [upload=273764]: Waiting for error group managing segments... Aug 19 16:44:41.244668372 (263h18m35.675748503s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.277280365 (263h18m35.708360496s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.328561665 (263h18m35.759641786s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.331747139 (263h18m35.76282727s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.357720320 (263h18m35.788800481s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.378190805 (263h18m35.809270936s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.486480333 (263h18m35.917560464s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:45:38.230570787 (263h19m32.661650918s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: expected piece hash; context deadline exceeded; EOF Aug 19 16:45:38.230979869 (263h19m32.66206s) [noise=true , address=77.90.31.18:4840 , piece_id=JWOETSFSZ3HELLZNLEASYNGW5OR64TMQVOFYYJAJVTHWCBHVTSLQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: read tcp 188.166.225.170:39946->77.90.31.18:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231017812 (263h19m32.662097943s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 188.166.225.170:44164->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231063489 (263h19m32.66214362s) [noise=true , address=45.91.116.14:4203 , piece_id=WRLBES3FI5EA5AQLZXJYGDROP3KBG3E2F22R4YD6A5KKI44PLV2Q, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 188.166.225.170:51738->45.91.116.14:4203: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231090952 (263h19m32.662171083s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: read tcp 188.166.225.170:57158->77.90.31.15:4707: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231110719 (263h19m32.66219085s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:50784->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231140165 (263h19m32.662220296s) [noise=true , address=77.90.31.16:4749 , piece_id=XCPXVBYLQH345FPIW4KDNLBFUCDDEW2H4B25WIXGOO5FVMAIVKPQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: read tcp 188.166.225.170:51668->77.90.31.16:4749: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231144814 (263h19m32.662224945s) [noise=true , address=77.90.31.19:4865 , piece_id=B3HUFIJZKVZJH4OBBO2DD7S3I7HX4LO2IQ74XJPO4DTW4QER35NA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1bguBghQonV5sTHaWteyUEnpp5KNzBKbvBpbvrUNFDCveHN77X, address:77.90.31.19:4865): protocol: read tcp 188.166.225.170:39398->77.90.31.19:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231177167 (263h19m32.662257298s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: read tcp 188.166.225.170:58526->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231184571 (263h19m32.662264702s) [noise=true , address=45.91.116.12:4236 , piece_id=LCRQS2GOSHOSLZBGQTGB3Q3PA4VN7KXNW66TDZ5O424TPJWTQE3A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH, address:45.91.116.12:4236): protocol: read tcp 188.166.225.170:57166->45.91.116.12:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231189770 (263h19m32.662269901s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 188.166.225.170:54242->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231215850 (263h19m32.662295981s) [seg_pos={0 0}, upload=273764]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 19 16:45:38.459100173 (263h19m32.890180304s) [upload=273764]: Done uploading segments... Aug 19 16:45:38.459128316 (263h19m32.890208447s) [upload=273764]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.812946042s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X...X..............X......X.XX........................X..] duration: 5.978611813s FAIL (2025-08-19 17:10:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== Aug 19 17:09:15.684096389 (263h43m10.11517652s) [upload=274176]: Starting upload Aug 19 17:09:15.684143008 (263h43m10.115223139s) [upload=274176]: Done starting upload Aug 19 17:09:15.684256185 (263h43m10.115336306s) [upload=274176]: Uploading segments... Aug 19 17:09:15.684468862 (263h43m10.115548993s) [upload=274176]: Got next segment. Inline: false Aug 19 17:09:15.684504950 (263h43m10.115585082s) [upload=274176]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:09:15.684509770 (263h43m10.115589901s) [upload=274176]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:09:16.135567491 (263h43m10.566647622s) [seg_pos={0 0}, upload=274176]: Begin upload segment... Aug 19 17:09:16.135613589 (263h43m10.56669372s) [seg_pos={0 0}, upload=274176]: Done begin upload segment. Aug 19 17:09:16.135636573 (263h43m10.566716695s) [noise=true , address=104.36.82.5:4853 , piece_id=K6TRSISY75NXYJNAAEALQ7W4RA26P5TGIVRMHX5SGOIF2GJ7X3GQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.135699574 (263h43m10.566779705s) [noise=true , address=104.36.82.60:4341 , piece_id=5NWVHAFJXPCBGPMRT42H4C3LPE3XKBDKPSX44D3DOWF6XXXJ5LFA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136028283 (263h43m10.567108404s) [noise=true , address=79.127.220.98:4432 , piece_id=4VA6KKCGA5GM55N3SD7FNUILD5XWYHE5VFRBFDIBAGVEK2VHQELA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136206604 (263h43m10.567286735s) [noise=true , address=104.36.82.22:4538 , piece_id=YUOOOPSZLQRNVU557APDO5FTOCY4FEYAHCBZQEE6YWMBPK5ZJYNA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136295745 (263h43m10.567375876s) [noise=true , address=77.90.31.11:4564 , piece_id=TX3C4VLEEOJOHBN7VDK4KIROG42PDDCB3SK26FZA7PM2455YZCMA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136640434 (263h43m10.567720565s) [noise=true , address=104.36.82.16:4315 , piece_id=P3SYICVJ5HSVY4MM62FDXQXJARZLIOQYNPBCT2SKYFPYZ3G3MSWQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136649702 (263h43m10.567729833s) [noise=true , address=104.36.82.44:4874 , piece_id=D3L2BM57MTGKIWKRSSEL7DRXUOK7LLKHL7M3Y3WAFCBILVDEV24Q, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136950888 (263h43m10.568031019s) [noise=true , address=104.36.82.35:4605 , piece_id=ORHH35ESP7KPGGDRD7ANECFHW7YTA2VFZTB5EMRCBQX5EWB7ZU4Q, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136961468 (263h43m10.568041599s) [noise=true , address=77.90.31.10:4535 , piece_id=N6LDVXERZAIR3EZ2TAMFQBM5Y565FMSP44I5KAWC32DTIOOM3FIQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137201027 (263h43m10.568281158s) [noise=true , address=77.90.31.12:4611 , piece_id=BANXKRHIMMOQFBSC4FW2E3SMNXCQERG2OTD7GXMM56T3X6JPNQGQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137260470 (263h43m10.568340601s) [noise=true , address=104.36.82.11:4134 , piece_id=BSGC4ADI6LMK3E2YEYPWFMYS3GC5VA42YM22VIJEGZ2JANYMLBYQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137463418 (263h43m10.568543549s) [noise=true , address=45.91.116.10:4110 , piece_id=KCZZOHPLSRNA45VXOMW7AKUYMRRFRUA52KSOBHNDRJPCB3RMV5HA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137553992 (263h43m10.568634123s) [noise=true , address=104.36.82.55:4224 , piece_id=PGZXUHGOONWDPIB4ZHAKLL4F2UWMZJPAL7BOTXIGSQWESLUPWGQA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137682447 (263h43m10.568762578s) [noise=true , address=45.91.116.12:4238 , piece_id=JAUKDCYIMCIOX4EFHQMCNCBLVZEIBKODPBL5QVBPOJIB554KLNRQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137883032 (263h43m10.568963163s) [noise=true , address=77.90.31.16:4743 , piece_id=YKPMLYPRMAZEBX7ORWEICH5AWKVS2J74D25W4ANIJJOUIBX5UBRA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137912418 (263h43m10.568992539s) [noise=true , address=104.36.82.23:4455 , piece_id=L2M5P5LRKVLVVDSYIOBV4SR3EAH2UO2JUZBGKZKIWBUIGFZPVEOA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.138136216 (263h43m10.569216347s) [noise=true , address=104.36.82.7:4930 , piece_id=EGTR52TZDAVTPPRCJLK4GOUXUWW2U7XJMQOV5P3JCKAKOK7GG5JQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.138374362 (263h43m10.569454493s) [noise=true , address=77.90.31.18:4839 , piece_id=BG553GCZSJEH5ORINSIQ7IUYU3UBQ3WO73ZAGRB7W45HOASJMBVQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:17.616571604 (263h43m12.047651765s) [upload=274176]: Next returned nil segment Aug 19 17:09:17.616591412 (263h43m12.047671543s) [upload=274176]: Waiting for error group managing segments... Aug 19 17:09:18.008239751 (263h43m12.439319882s) [noise=true , address=79.127.220.98:4432 , piece_id=4VA6KKCGA5GM55N3SD7FNUILD5XWYHE5VFRBFDIBAGVEK2VHQELA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.039911371 (263h43m12.470991502s) [noise=true , address=45.91.116.12:4238 , piece_id=JAUKDCYIMCIOX4EFHQMCNCBLVZEIBKODPBL5QVBPOJIB554KLNRQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.065695686 (263h43m12.496775817s) [noise=true , address=77.90.31.16:4743 , piece_id=YKPMLYPRMAZEBX7ORWEICH5AWKVS2J74D25W4ANIJJOUIBX5UBRA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.080085439 (263h43m12.51116557s) [noise=true , address=45.91.116.10:4110 , piece_id=KCZZOHPLSRNA45VXOMW7AKUYMRRFRUA52KSOBHNDRJPCB3RMV5HA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.131454157 (263h43m12.562534288s) [noise=true , address=104.36.82.11:4134 , piece_id=BSGC4ADI6LMK3E2YEYPWFMYS3GC5VA42YM22VIJEGZ2JANYMLBYQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.143557214 (263h43m12.574637345s) [noise=true , address=104.36.82.7:4930 , piece_id=EGTR52TZDAVTPPRCJLK4GOUXUWW2U7XJMQOV5P3JCKAKOK7GG5JQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.156135631 (263h43m12.587215762s) [noise=true , address=77.90.31.12:4611 , piece_id=BANXKRHIMMOQFBSC4FW2E3SMNXCQERG2OTD7GXMM56T3X6JPNQGQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.159345252 (263h43m12.590425373s) [noise=true , address=104.36.82.55:4224 , piece_id=PGZXUHGOONWDPIB4ZHAKLL4F2UWMZJPAL7BOTXIGSQWESLUPWGQA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.169106219 (263h43m12.60018635s) [noise=true , address=104.36.82.22:4538 , piece_id=YUOOOPSZLQRNVU557APDO5FTOCY4FEYAHCBZQEE6YWMBPK5ZJYNA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.188273593 (263h43m12.619353724s) [noise=true , address=104.36.82.35:4605 , piece_id=ORHH35ESP7KPGGDRD7ANECFHW7YTA2VFZTB5EMRCBQX5EWB7ZU4Q, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.204131846 (263h43m12.635211977s) [noise=true , address=104.36.82.16:4315 , piece_id=P3SYICVJ5HSVY4MM62FDXQXJARZLIOQYNPBCT2SKYFPYZ3G3MSWQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.217541965 (263h43m12.648622096s) [noise=true , address=104.36.82.60:4341 , piece_id=5NWVHAFJXPCBGPMRT42H4C3LPE3XKBDKPSX44D3DOWF6XXXJ5LFA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.222565827 (263h43m12.653645948s) [noise=true , address=104.36.82.23:4455 , piece_id=L2M5P5LRKVLVVDSYIOBV4SR3EAH2UO2JUZBGKZKIWBUIGFZPVEOA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.377619011 (263h43m12.808699132s) [noise=true , address=104.36.82.44:4874 , piece_id=D3L2BM57MTGKIWKRSSEL7DRXUOK7LLKHL7M3Y3WAFCBILVDEV24Q, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.377646604 (263h43m12.808726725s) [seg_pos={0 0}, upload=274176]: Segment reached optimal threshold of 14 pieces. Aug 19 17:09:18.377754100 (263h43m12.808834231s) [noise=true , address=104.36.82.5:4853 , piece_id=K6TRSISY75NXYJNAAEALQ7W4RA26P5TGIVRMHX5SGOIF2GJ7X3GQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377895570 (263h43m12.808975701s) [noise=true , address=77.90.31.18:4839 , piece_id=BG553GCZSJEH5ORINSIQ7IUYU3UBQ3WO73ZAGRB7W45HOASJMBVQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gmWwyaQ6zM6QdayBFpXdE9VSE1aaQcBHFcukJoASjjHWCDsNx): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377915529 (263h43m12.80899566s) [noise=true , address=77.90.31.11:4564 , piece_id=TX3C4VLEEOJOHBN7VDK4KIROG42PDDCB3SK26FZA7PM2455YZCMA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377974832 (263h43m12.809054963s) [noise=true , address=77.90.31.10:4535 , piece_id=N6LDVXERZAIR3EZ2TAMFQBM5Y565FMSP44I5KAWC32DTIOOM3FIQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377999008 (263h43m12.80907914s) [seg_pos={0 0}, upload=274176]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 17:09:18.378145599 (263h43m12.80922573s) [upload=274176]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 17:09:18.378150488 (263h43m12.809230619s) [upload=274176]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:09:18.983488776 (263h43m13.414568907s) [upload=274176]: Done uploading segments... Aug 19 17:09:18.983507222 (263h43m13.414587353s) [upload=274176]: Upload finished. err: <nil> Duration: 5.978629869s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................X....XX.............................] duration: 5.832658849s FAIL (2025-08-19 17:10:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 19 16:49:12.801154313 (263h23m7.232234445s) [upload=273839]: Starting upload Aug 19 16:49:12.801215942 (263h23m7.232296073s) [upload=273839]: Done starting upload Aug 19 16:49:12.801330351 (263h23m7.232410482s) [upload=273839]: Uploading segments... Aug 19 16:49:12.801425372 (263h23m7.232505503s) [upload=273839]: Got next segment. Inline: false Aug 19 16:49:12.801455880 (263h23m7.232536012s) [upload=273839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:49:12.801461982 (263h23m7.232542113s) [upload=273839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:49:13.036708228 (263h23m7.467788359s) [seg_pos={0 0}, upload=273839]: Begin upload segment... Aug 19 16:49:13.036749197 (263h23m7.467829328s) [seg_pos={0 0}, upload=273839]: Done begin upload segment. Aug 19 16:49:13.036771971 (263h23m7.467852092s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.036780256 (263h23m7.467860437s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037135917 (263h23m7.468216048s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037326592 (263h23m7.468406723s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037403259 (263h23m7.46848339s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037667814 (263h23m7.468747945s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037761764 (263h23m7.468841896s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037943331 (263h23m7.469023462s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038070565 (263h23m7.469150696s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038168622 (263h23m7.469248753s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038300104 (263h23m7.469380235s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038396789 (263h23m7.46947692s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038622241 (263h23m7.469702362s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038679350 (263h23m7.469759481s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038854735 (263h23m7.469934856s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.039048045 (263h23m7.470128166s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:14.749064365 (263h23m9.180144496s) [upload=273839]: Next returned nil segment Aug 19 16:49:14.749117217 (263h23m9.180197348s) [upload=273839]: Waiting for error group managing segments... Aug 19 16:49:15.090317863 (263h23m9.521397994s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.132109007 (263h23m9.563189138s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.222090156 (263h23m9.653170287s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.328041780 (263h23m9.759121911s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.353998781 (263h23m9.785078912s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:36.709543777 (263h23m31.140623898s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:37.041062020 (263h23m31.472142141s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.183707791 (263h23m32.614787922s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.694075084 (263h23m33.125155215s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.716788163 (263h23m34.147868294s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.851890717 (263h23m34.282970848s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:40.293306092 (263h23m34.724386243s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:42.156929229 (263h23m36.58800936s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157075889 (263h23m36.58815601s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157126075 (263h23m36.588206206s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection Aug 19 16:49:42.157170951 (263h23m36.588251082s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: read tcp 188.166.225.170:34242->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:42.157215747 (263h23m36.588295878s) [seg_pos={0 0}, upload=273839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 19 16:49:42.385478364 (263h23m36.816558495s) [upload=273839]: Done uploading segments... Aug 19 16:49:42.385503091 (263h23m36.816583212s) [upload=273839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.832706931s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........................X...X...............................] duration: 6.485809532s FAIL (2025-08-19 17:10:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.485833749s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......X..........................................X.........] duration: 1.989227912s FAIL (2025-08-19 17:10:22 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 17:04:06.343826144 (263h38m0.774906265s) [upload=274088]: Starting upload Aug 19 17:04:06.343875278 (263h38m0.774955409s) [upload=274088]: Done starting upload Aug 19 17:04:06.343971703 (263h38m0.775051834s) [upload=274088]: Uploading segments... Aug 19 17:04:06.344028321 (263h38m0.775108453s) [upload=274088]: Got next segment. Inline: true Aug 19 17:04:06.344033120 (263h38m0.775113252s) [upload=274088]: Waiting for error group managing segments... Aug 19 17:04:06.344068187 (263h38m0.775148318s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:04:06.344071964 (263h38m0.775152095s) [upload=274088]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:04:06.344074499 (263h38m0.77515462s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:04:06.653287654 (263h38m1.084367785s) [upload=274088]: Done uploading segments... Aug 19 17:04:06.653300519 (263h38m1.08438065s) [upload=274088]: Upload finished. err: <nil> Duration:* 1.989263759s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [X...X...X..............X......X.XX........................X.] duration: 5.959941196s FAIL (2025-08-19 17:10:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== Aug 19 17:09:15.684096389 (263h43m10.11517652s) [upload=274176]: Starting upload Aug 19 17:09:15.684143008 (263h43m10.115223139s) [upload=274176]: Done starting upload Aug 19 17:09:15.684256185 (263h43m10.115336306s) [upload=274176]: Uploading segments... Aug 19 17:09:15.684468862 (263h43m10.115548993s) [upload=274176]: Got next segment. Inline: false Aug 19 17:09:15.684504950 (263h43m10.115585082s) [upload=274176]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:09:15.684509770 (263h43m10.115589901s) [upload=274176]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:09:16.135567491 (263h43m10.566647622s) [seg_pos={0 0}, upload=274176]: Begin upload segment... Aug 19 17:09:16.135613589 (263h43m10.56669372s) [seg_pos={0 0}, upload=274176]: Done begin upload segment. Aug 19 17:09:16.135636573 (263h43m10.566716695s) [noise=true , address=104.36.82.5:4853 , piece_id=K6TRSISY75NXYJNAAEALQ7W4RA26P5TGIVRMHX5SGOIF2GJ7X3GQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.135699574 (263h43m10.566779705s) [noise=true , address=104.36.82.60:4341 , piece_id=5NWVHAFJXPCBGPMRT42H4C3LPE3XKBDKPSX44D3DOWF6XXXJ5LFA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136028283 (263h43m10.567108404s) [noise=true , address=79.127.220.98:4432 , piece_id=4VA6KKCGA5GM55N3SD7FNUILD5XWYHE5VFRBFDIBAGVEK2VHQELA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136206604 (263h43m10.567286735s) [noise=true , address=104.36.82.22:4538 , piece_id=YUOOOPSZLQRNVU557APDO5FTOCY4FEYAHCBZQEE6YWMBPK5ZJYNA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136295745 (263h43m10.567375876s) [noise=true , address=77.90.31.11:4564 , piece_id=TX3C4VLEEOJOHBN7VDK4KIROG42PDDCB3SK26FZA7PM2455YZCMA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136640434 (263h43m10.567720565s) [noise=true , address=104.36.82.16:4315 , piece_id=P3SYICVJ5HSVY4MM62FDXQXJARZLIOQYNPBCT2SKYFPYZ3G3MSWQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136649702 (263h43m10.567729833s) [noise=true , address=104.36.82.44:4874 , piece_id=D3L2BM57MTGKIWKRSSEL7DRXUOK7LLKHL7M3Y3WAFCBILVDEV24Q, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136950888 (263h43m10.568031019s) [noise=true , address=104.36.82.35:4605 , piece_id=ORHH35ESP7KPGGDRD7ANECFHW7YTA2VFZTB5EMRCBQX5EWB7ZU4Q, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136961468 (263h43m10.568041599s) [noise=true , address=77.90.31.10:4535 , piece_id=N6LDVXERZAIR3EZ2TAMFQBM5Y565FMSP44I5KAWC32DTIOOM3FIQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137201027 (263h43m10.568281158s) [noise=true , address=77.90.31.12:4611 , piece_id=BANXKRHIMMOQFBSC4FW2E3SMNXCQERG2OTD7GXMM56T3X6JPNQGQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137260470 (263h43m10.568340601s) [noise=true , address=104.36.82.11:4134 , piece_id=BSGC4ADI6LMK3E2YEYPWFMYS3GC5VA42YM22VIJEGZ2JANYMLBYQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137463418 (263h43m10.568543549s) [noise=true , address=45.91.116.10:4110 , piece_id=KCZZOHPLSRNA45VXOMW7AKUYMRRFRUA52KSOBHNDRJPCB3RMV5HA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137553992 (263h43m10.568634123s) [noise=true , address=104.36.82.55:4224 , piece_id=PGZXUHGOONWDPIB4ZHAKLL4F2UWMZJPAL7BOTXIGSQWESLUPWGQA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137682447 (263h43m10.568762578s) [noise=true , address=45.91.116.12:4238 , piece_id=JAUKDCYIMCIOX4EFHQMCNCBLVZEIBKODPBL5QVBPOJIB554KLNRQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137883032 (263h43m10.568963163s) [noise=true , address=77.90.31.16:4743 , piece_id=YKPMLYPRMAZEBX7ORWEICH5AWKVS2J74D25W4ANIJJOUIBX5UBRA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137912418 (263h43m10.568992539s) [noise=true , address=104.36.82.23:4455 , piece_id=L2M5P5LRKVLVVDSYIOBV4SR3EAH2UO2JUZBGKZKIWBUIGFZPVEOA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.138136216 (263h43m10.569216347s) [noise=true , address=104.36.82.7:4930 , piece_id=EGTR52TZDAVTPPRCJLK4GOUXUWW2U7XJMQOV5P3JCKAKOK7GG5JQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.138374362 (263h43m10.569454493s) [noise=true , address=77.90.31.18:4839 , piece_id=BG553GCZSJEH5ORINSIQ7IUYU3UBQ3WO73ZAGRB7W45HOASJMBVQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:17.616571604 (263h43m12.047651765s) [upload=274176]: Next returned nil segment Aug 19 17:09:17.616591412 (263h43m12.047671543s) [upload=274176]: Waiting for error group managing segments... Aug 19 17:09:18.008239751 (263h43m12.439319882s) [noise=true , address=79.127.220.98:4432 , piece_id=4VA6KKCGA5GM55N3SD7FNUILD5XWYHE5VFRBFDIBAGVEK2VHQELA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.039911371 (263h43m12.470991502s) [noise=true , address=45.91.116.12:4238 , piece_id=JAUKDCYIMCIOX4EFHQMCNCBLVZEIBKODPBL5QVBPOJIB554KLNRQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.065695686 (263h43m12.496775817s) [noise=true , address=77.90.31.16:4743 , piece_id=YKPMLYPRMAZEBX7ORWEICH5AWKVS2J74D25W4ANIJJOUIBX5UBRA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.080085439 (263h43m12.51116557s) [noise=true , address=45.91.116.10:4110 , piece_id=KCZZOHPLSRNA45VXOMW7AKUYMRRFRUA52KSOBHNDRJPCB3RMV5HA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.131454157 (263h43m12.562534288s) [noise=true , address=104.36.82.11:4134 , piece_id=BSGC4ADI6LMK3E2YEYPWFMYS3GC5VA42YM22VIJEGZ2JANYMLBYQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.143557214 (263h43m12.574637345s) [noise=true , address=104.36.82.7:4930 , piece_id=EGTR52TZDAVTPPRCJLK4GOUXUWW2U7XJMQOV5P3JCKAKOK7GG5JQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.156135631 (263h43m12.587215762s) [noise=true , address=77.90.31.12:4611 , piece_id=BANXKRHIMMOQFBSC4FW2E3SMNXCQERG2OTD7GXMM56T3X6JPNQGQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.159345252 (263h43m12.590425373s) [noise=true , address=104.36.82.55:4224 , piece_id=PGZXUHGOONWDPIB4ZHAKLL4F2UWMZJPAL7BOTXIGSQWESLUPWGQA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.169106219 (263h43m12.60018635s) [noise=true , address=104.36.82.22:4538 , piece_id=YUOOOPSZLQRNVU557APDO5FTOCY4FEYAHCBZQEE6YWMBPK5ZJYNA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.188273593 (263h43m12.619353724s) [noise=true , address=104.36.82.35:4605 , piece_id=ORHH35ESP7KPGGDRD7ANECFHW7YTA2VFZTB5EMRCBQX5EWB7ZU4Q, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.204131846 (263h43m12.635211977s) [noise=true , address=104.36.82.16:4315 , piece_id=P3SYICVJ5HSVY4MM62FDXQXJARZLIOQYNPBCT2SKYFPYZ3G3MSWQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.217541965 (263h43m12.648622096s) [noise=true , address=104.36.82.60:4341 , piece_id=5NWVHAFJXPCBGPMRT42H4C3LPE3XKBDKPSX44D3DOWF6XXXJ5LFA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.222565827 (263h43m12.653645948s) [noise=true , address=104.36.82.23:4455 , piece_id=L2M5P5LRKVLVVDSYIOBV4SR3EAH2UO2JUZBGKZKIWBUIGFZPVEOA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.377619011 (263h43m12.808699132s) [noise=true , address=104.36.82.44:4874 , piece_id=D3L2BM57MTGKIWKRSSEL7DRXUOK7LLKHL7M3Y3WAFCBILVDEV24Q, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.377646604 (263h43m12.808726725s) [seg_pos={0 0}, upload=274176]: Segment reached optimal threshold of 14 pieces. Aug 19 17:09:18.377754100 (263h43m12.808834231s) [noise=true , address=104.36.82.5:4853 , piece_id=K6TRSISY75NXYJNAAEALQ7W4RA26P5TGIVRMHX5SGOIF2GJ7X3GQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377895570 (263h43m12.808975701s) [noise=true , address=77.90.31.18:4839 , piece_id=BG553GCZSJEH5ORINSIQ7IUYU3UBQ3WO73ZAGRB7W45HOASJMBVQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gmWwyaQ6zM6QdayBFpXdE9VSE1aaQcBHFcukJoASjjHWCDsNx): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377915529 (263h43m12.80899566s) [noise=true , address=77.90.31.11:4564 , piece_id=TX3C4VLEEOJOHBN7VDK4KIROG42PDDCB3SK26FZA7PM2455YZCMA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377974832 (263h43m12.809054963s) [noise=true , address=77.90.31.10:4535 , piece_id=N6LDVXERZAIR3EZ2TAMFQBM5Y565FMSP44I5KAWC32DTIOOM3FIQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377999008 (263h43m12.80907914s) [seg_pos={0 0}, upload=274176]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 17:09:18.378145599 (263h43m12.80922573s) [upload=274176]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 17:09:18.378150488 (263h43m12.809230619s) [upload=274176]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:09:18.983488776 (263h43m13.414568907s) [upload=274176]: Done uploading segments... Aug 19 17:09:18.983507222 (263h43m13.414587353s) [upload=274176]: Upload finished. err: <nil> Duration: 5.959971785s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........................X...X..............................] duration: 6.341507131s FAIL (2025-08-19 17:10:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.341519204s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)