Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X....XX...............................] duration: 5.897478667s FAIL (2025-08-19 17:12:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 19 16:49:12.801154313 (263h23m7.232234445s) [upload=273839]: Starting upload Aug 19 16:49:12.801215942 (263h23m7.232296073s) [upload=273839]: Done starting upload Aug 19 16:49:12.801330351 (263h23m7.232410482s) [upload=273839]: Uploading segments... Aug 19 16:49:12.801425372 (263h23m7.232505503s) [upload=273839]: Got next segment. Inline: false Aug 19 16:49:12.801455880 (263h23m7.232536012s) [upload=273839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:49:12.801461982 (263h23m7.232542113s) [upload=273839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:49:13.036708228 (263h23m7.467788359s) [seg_pos={0 0}, upload=273839]: Begin upload segment... Aug 19 16:49:13.036749197 (263h23m7.467829328s) [seg_pos={0 0}, upload=273839]: Done begin upload segment. Aug 19 16:49:13.036771971 (263h23m7.467852092s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.036780256 (263h23m7.467860437s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037135917 (263h23m7.468216048s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037326592 (263h23m7.468406723s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037403259 (263h23m7.46848339s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037667814 (263h23m7.468747945s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037761764 (263h23m7.468841896s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037943331 (263h23m7.469023462s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038070565 (263h23m7.469150696s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038168622 (263h23m7.469248753s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038300104 (263h23m7.469380235s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038396789 (263h23m7.46947692s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038622241 (263h23m7.469702362s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038679350 (263h23m7.469759481s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038854735 (263h23m7.469934856s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.039048045 (263h23m7.470128166s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:14.749064365 (263h23m9.180144496s) [upload=273839]: Next returned nil segment Aug 19 16:49:14.749117217 (263h23m9.180197348s) [upload=273839]: Waiting for error group managing segments... Aug 19 16:49:15.090317863 (263h23m9.521397994s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.132109007 (263h23m9.563189138s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.222090156 (263h23m9.653170287s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.328041780 (263h23m9.759121911s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.353998781 (263h23m9.785078912s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:36.709543777 (263h23m31.140623898s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:37.041062020 (263h23m31.472142141s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.183707791 (263h23m32.614787922s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.694075084 (263h23m33.125155215s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.716788163 (263h23m34.147868294s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.851890717 (263h23m34.282970848s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:40.293306092 (263h23m34.724386243s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:42.156929229 (263h23m36.58800936s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157075889 (263h23m36.58815601s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157126075 (263h23m36.588206206s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection Aug 19 16:49:42.157170951 (263h23m36.588251082s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: read tcp 188.166.225.170:34242->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:42.157215747 (263h23m36.588295878s) [seg_pos={0 0}, upload=273839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 19 16:49:42.385478364 (263h23m36.816558495s) [upload=273839]: Done uploading segments... Aug 19 16:49:42.385503091 (263h23m36.816583212s) [upload=273839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.897511008s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....X..........................................X...........] duration: 1.998738161s FAIL (2025-08-19 17:12:20 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 17:04:06.343826144 (263h38m0.774906265s) [upload=274088]: Starting upload Aug 19 17:04:06.343875278 (263h38m0.774955409s) [upload=274088]: Done starting upload Aug 19 17:04:06.343971703 (263h38m0.775051834s) [upload=274088]: Uploading segments... Aug 19 17:04:06.344028321 (263h38m0.775108453s) [upload=274088]: Got next segment. Inline: true Aug 19 17:04:06.344033120 (263h38m0.775113252s) [upload=274088]: Waiting for error group managing segments... Aug 19 17:04:06.344068187 (263h38m0.775148318s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:04:06.344071964 (263h38m0.775152095s) [upload=274088]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:04:06.344074499 (263h38m0.77515462s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:04:06.653287654 (263h38m1.084367785s) [upload=274088]: Done uploading segments... Aug 19 17:04:06.653300519 (263h38m1.08438065s) [upload=274088]: Upload finished. err: <nil> Duration:* 1.998762107s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................X...X.................................] duration: 6.299057304s FAIL (2025-08-19 17:12:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.299078815s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.571802071s FAIL (2025-08-19 17:12:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 263h18m33.75518017s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324344006 (263h18m33.755424127s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324633220 (263h18m33.755713351s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324851286 (263h18m33.755931417s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325068942 (263h18m33.756149063s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325283613 (263h18m33.756363744s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325499857 (263h18m33.756579998s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325759794 (263h18m33.756839915s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326059428 (263h18m33.757139549s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326277565 (263h18m33.757357696s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326470835 (263h18m33.757550966s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326673753 (263h18m33.757753884s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326867664 (263h18m33.757947795s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:40.851885389 (263h18m35.28296552s) [upload=273764]: Next returned nil segment Aug 19 16:44:40.851903334 (263h18m35.282983465s) [upload=273764]: Waiting for error group managing segments... Aug 19 16:44:41.244668372 (263h18m35.675748503s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.277280365 (263h18m35.708360496s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.328561665 (263h18m35.759641786s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.331747139 (263h18m35.76282727s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.357720320 (263h18m35.788800481s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.378190805 (263h18m35.809270936s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.486480333 (263h18m35.917560464s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:45:38.230570787 (263h19m32.661650918s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: expected piece hash; context deadline exceeded; EOF Aug 19 16:45:38.230979869 (263h19m32.66206s) [noise=true , address=77.90.31.18:4840 , piece_id=JWOETSFSZ3HELLZNLEASYNGW5OR64TMQVOFYYJAJVTHWCBHVTSLQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: read tcp 188.166.225.170:39946->77.90.31.18:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231017812 (263h19m32.662097943s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 188.166.225.170:44164->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231063489 (263h19m32.66214362s) [noise=true , address=45.91.116.14:4203 , piece_id=WRLBES3FI5EA5AQLZXJYGDROP3KBG3E2F22R4YD6A5KKI44PLV2Q, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 188.166.225.170:51738->45.91.116.14:4203: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231090952 (263h19m32.662171083s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: read tcp 188.166.225.170:57158->77.90.31.15:4707: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231110719 (263h19m32.66219085s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:50784->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231140165 (263h19m32.662220296s) [noise=true , address=77.90.31.16:4749 , piece_id=XCPXVBYLQH345FPIW4KDNLBFUCDDEW2H4B25WIXGOO5FVMAIVKPQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: read tcp 188.166.225.170:51668->77.90.31.16:4749: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231144814 (263h19m32.662224945s) [noise=true , address=77.90.31.19:4865 , piece_id=B3HUFIJZKVZJH4OBBO2DD7S3I7HX4LO2IQ74XJPO4DTW4QER35NA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1bguBghQonV5sTHaWteyUEnpp5KNzBKbvBpbvrUNFDCveHN77X, address:77.90.31.19:4865): protocol: read tcp 188.166.225.170:39398->77.90.31.19:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231177167 (263h19m32.662257298s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: read tcp 188.166.225.170:58526->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231184571 (263h19m32.662264702s) [noise=true , address=45.91.116.12:4236 , piece_id=LCRQS2GOSHOSLZBGQTGB3Q3PA4VN7KXNW66TDZ5O424TPJWTQE3A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH, address:45.91.116.12:4236): protocol: read tcp 188.166.225.170:57166->45.91.116.12:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231189770 (263h19m32.662269901s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 188.166.225.170:54242->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231215850 (263h19m32.662295981s) [seg_pos={0 0}, upload=273764]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 19 16:45:38.459100173 (263h19m32.890180304s) [upload=273764]: Done uploading segments... Aug 19 16:45:38.459128316 (263h19m32.890208447s) [upload=273764]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.571825526s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X....X..............................] duration: 5.864693872s FAIL (2025-08-19 17:12:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.86471877s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X...X..............X......X.XX........................X...] duration: 6.003479339s FAIL (2025-08-19 17:11:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== Aug 19 17:09:15.684096389 (263h43m10.11517652s) [upload=274176]: Starting upload Aug 19 17:09:15.684143008 (263h43m10.115223139s) [upload=274176]: Done starting upload Aug 19 17:09:15.684256185 (263h43m10.115336306s) [upload=274176]: Uploading segments... Aug 19 17:09:15.684468862 (263h43m10.115548993s) [upload=274176]: Got next segment. Inline: false Aug 19 17:09:15.684504950 (263h43m10.115585082s) [upload=274176]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:09:15.684509770 (263h43m10.115589901s) [upload=274176]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:09:16.135567491 (263h43m10.566647622s) [seg_pos={0 0}, upload=274176]: Begin upload segment... Aug 19 17:09:16.135613589 (263h43m10.56669372s) [seg_pos={0 0}, upload=274176]: Done begin upload segment. Aug 19 17:09:16.135636573 (263h43m10.566716695s) [noise=true , address=104.36.82.5:4853 , piece_id=K6TRSISY75NXYJNAAEALQ7W4RA26P5TGIVRMHX5SGOIF2GJ7X3GQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.135699574 (263h43m10.566779705s) [noise=true , address=104.36.82.60:4341 , piece_id=5NWVHAFJXPCBGPMRT42H4C3LPE3XKBDKPSX44D3DOWF6XXXJ5LFA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136028283 (263h43m10.567108404s) [noise=true , address=79.127.220.98:4432 , piece_id=4VA6KKCGA5GM55N3SD7FNUILD5XWYHE5VFRBFDIBAGVEK2VHQELA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136206604 (263h43m10.567286735s) [noise=true , address=104.36.82.22:4538 , piece_id=YUOOOPSZLQRNVU557APDO5FTOCY4FEYAHCBZQEE6YWMBPK5ZJYNA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136295745 (263h43m10.567375876s) [noise=true , address=77.90.31.11:4564 , piece_id=TX3C4VLEEOJOHBN7VDK4KIROG42PDDCB3SK26FZA7PM2455YZCMA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136640434 (263h43m10.567720565s) [noise=true , address=104.36.82.16:4315 , piece_id=P3SYICVJ5HSVY4MM62FDXQXJARZLIOQYNPBCT2SKYFPYZ3G3MSWQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136649702 (263h43m10.567729833s) [noise=true , address=104.36.82.44:4874 , piece_id=D3L2BM57MTGKIWKRSSEL7DRXUOK7LLKHL7M3Y3WAFCBILVDEV24Q, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136950888 (263h43m10.568031019s) [noise=true , address=104.36.82.35:4605 , piece_id=ORHH35ESP7KPGGDRD7ANECFHW7YTA2VFZTB5EMRCBQX5EWB7ZU4Q, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136961468 (263h43m10.568041599s) [noise=true , address=77.90.31.10:4535 , piece_id=N6LDVXERZAIR3EZ2TAMFQBM5Y565FMSP44I5KAWC32DTIOOM3FIQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137201027 (263h43m10.568281158s) [noise=true , address=77.90.31.12:4611 , piece_id=BANXKRHIMMOQFBSC4FW2E3SMNXCQERG2OTD7GXMM56T3X6JPNQGQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137260470 (263h43m10.568340601s) [noise=true , address=104.36.82.11:4134 , piece_id=BSGC4ADI6LMK3E2YEYPWFMYS3GC5VA42YM22VIJEGZ2JANYMLBYQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137463418 (263h43m10.568543549s) [noise=true , address=45.91.116.10:4110 , piece_id=KCZZOHPLSRNA45VXOMW7AKUYMRRFRUA52KSOBHNDRJPCB3RMV5HA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137553992 (263h43m10.568634123s) [noise=true , address=104.36.82.55:4224 , piece_id=PGZXUHGOONWDPIB4ZHAKLL4F2UWMZJPAL7BOTXIGSQWESLUPWGQA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137682447 (263h43m10.568762578s) [noise=true , address=45.91.116.12:4238 , piece_id=JAUKDCYIMCIOX4EFHQMCNCBLVZEIBKODPBL5QVBPOJIB554KLNRQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137883032 (263h43m10.568963163s) [noise=true , address=77.90.31.16:4743 , piece_id=YKPMLYPRMAZEBX7ORWEICH5AWKVS2J74D25W4ANIJJOUIBX5UBRA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137912418 (263h43m10.568992539s) [noise=true , address=104.36.82.23:4455 , piece_id=L2M5P5LRKVLVVDSYIOBV4SR3EAH2UO2JUZBGKZKIWBUIGFZPVEOA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.138136216 (263h43m10.569216347s) [noise=true , address=104.36.82.7:4930 , piece_id=EGTR52TZDAVTPPRCJLK4GOUXUWW2U7XJMQOV5P3JCKAKOK7GG5JQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.138374362 (263h43m10.569454493s) [noise=true , address=77.90.31.18:4839 , piece_id=BG553GCZSJEH5ORINSIQ7IUYU3UBQ3WO73ZAGRB7W45HOASJMBVQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:17.616571604 (263h43m12.047651765s) [upload=274176]: Next returned nil segment Aug 19 17:09:17.616591412 (263h43m12.047671543s) [upload=274176]: Waiting for error group managing segments... Aug 19 17:09:18.008239751 (263h43m12.439319882s) [noise=true , address=79.127.220.98:4432 , piece_id=4VA6KKCGA5GM55N3SD7FNUILD5XWYHE5VFRBFDIBAGVEK2VHQELA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.039911371 (263h43m12.470991502s) [noise=true , address=45.91.116.12:4238 , piece_id=JAUKDCYIMCIOX4EFHQMCNCBLVZEIBKODPBL5QVBPOJIB554KLNRQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.065695686 (263h43m12.496775817s) [noise=true , address=77.90.31.16:4743 , piece_id=YKPMLYPRMAZEBX7ORWEICH5AWKVS2J74D25W4ANIJJOUIBX5UBRA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.080085439 (263h43m12.51116557s) [noise=true , address=45.91.116.10:4110 , piece_id=KCZZOHPLSRNA45VXOMW7AKUYMRRFRUA52KSOBHNDRJPCB3RMV5HA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.131454157 (263h43m12.562534288s) [noise=true , address=104.36.82.11:4134 , piece_id=BSGC4ADI6LMK3E2YEYPWFMYS3GC5VA42YM22VIJEGZ2JANYMLBYQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.143557214 (263h43m12.574637345s) [noise=true , address=104.36.82.7:4930 , piece_id=EGTR52TZDAVTPPRCJLK4GOUXUWW2U7XJMQOV5P3JCKAKOK7GG5JQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.156135631 (263h43m12.587215762s) [noise=true , address=77.90.31.12:4611 , piece_id=BANXKRHIMMOQFBSC4FW2E3SMNXCQERG2OTD7GXMM56T3X6JPNQGQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.159345252 (263h43m12.590425373s) [noise=true , address=104.36.82.55:4224 , piece_id=PGZXUHGOONWDPIB4ZHAKLL4F2UWMZJPAL7BOTXIGSQWESLUPWGQA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.169106219 (263h43m12.60018635s) [noise=true , address=104.36.82.22:4538 , piece_id=YUOOOPSZLQRNVU557APDO5FTOCY4FEYAHCBZQEE6YWMBPK5ZJYNA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.188273593 (263h43m12.619353724s) [noise=true , address=104.36.82.35:4605 , piece_id=ORHH35ESP7KPGGDRD7ANECFHW7YTA2VFZTB5EMRCBQX5EWB7ZU4Q, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.204131846 (263h43m12.635211977s) [noise=true , address=104.36.82.16:4315 , piece_id=P3SYICVJ5HSVY4MM62FDXQXJARZLIOQYNPBCT2SKYFPYZ3G3MSWQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.217541965 (263h43m12.648622096s) [noise=true , address=104.36.82.60:4341 , piece_id=5NWVHAFJXPCBGPMRT42H4C3LPE3XKBDKPSX44D3DOWF6XXXJ5LFA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.222565827 (263h43m12.653645948s) [noise=true , address=104.36.82.23:4455 , piece_id=L2M5P5LRKVLVVDSYIOBV4SR3EAH2UO2JUZBGKZKIWBUIGFZPVEOA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.377619011 (263h43m12.808699132s) [noise=true , address=104.36.82.44:4874 , piece_id=D3L2BM57MTGKIWKRSSEL7DRXUOK7LLKHL7M3Y3WAFCBILVDEV24Q, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.377646604 (263h43m12.808726725s) [seg_pos={0 0}, upload=274176]: Segment reached optimal threshold of 14 pieces. Aug 19 17:09:18.377754100 (263h43m12.808834231s) [noise=true , address=104.36.82.5:4853 , piece_id=K6TRSISY75NXYJNAAEALQ7W4RA26P5TGIVRMHX5SGOIF2GJ7X3GQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377895570 (263h43m12.808975701s) [noise=true , address=77.90.31.18:4839 , piece_id=BG553GCZSJEH5ORINSIQ7IUYU3UBQ3WO73ZAGRB7W45HOASJMBVQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gmWwyaQ6zM6QdayBFpXdE9VSE1aaQcBHFcukJoASjjHWCDsNx): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377915529 (263h43m12.80899566s) [noise=true , address=77.90.31.11:4564 , piece_id=TX3C4VLEEOJOHBN7VDK4KIROG42PDDCB3SK26FZA7PM2455YZCMA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377974832 (263h43m12.809054963s) [noise=true , address=77.90.31.10:4535 , piece_id=N6LDVXERZAIR3EZ2TAMFQBM5Y565FMSP44I5KAWC32DTIOOM3FIQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377999008 (263h43m12.80907914s) [seg_pos={0 0}, upload=274176]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 17:09:18.378145599 (263h43m12.80922573s) [upload=274176]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 17:09:18.378150488 (263h43m12.809230619s) [upload=274176]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:09:18.983488776 (263h43m13.414568907s) [upload=274176]: Done uploading segments... Aug 19 17:09:18.983507222 (263h43m13.414587353s) [upload=274176]: Upload finished. err: <nil> Duration: 6.003558309s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.617678434s FAIL (2025-08-19 17:11:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 263h18m33.75518017s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324344006 (263h18m33.755424127s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324633220 (263h18m33.755713351s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324851286 (263h18m33.755931417s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325068942 (263h18m33.756149063s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325283613 (263h18m33.756363744s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325499857 (263h18m33.756579998s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325759794 (263h18m33.756839915s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326059428 (263h18m33.757139549s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326277565 (263h18m33.757357696s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326470835 (263h18m33.757550966s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326673753 (263h18m33.757753884s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326867664 (263h18m33.757947795s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:40.851885389 (263h18m35.28296552s) [upload=273764]: Next returned nil segment Aug 19 16:44:40.851903334 (263h18m35.282983465s) [upload=273764]: Waiting for error group managing segments... Aug 19 16:44:41.244668372 (263h18m35.675748503s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.277280365 (263h18m35.708360496s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.328561665 (263h18m35.759641786s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.331747139 (263h18m35.76282727s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.357720320 (263h18m35.788800481s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.378190805 (263h18m35.809270936s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.486480333 (263h18m35.917560464s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:45:38.230570787 (263h19m32.661650918s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: expected piece hash; context deadline exceeded; EOF Aug 19 16:45:38.230979869 (263h19m32.66206s) [noise=true , address=77.90.31.18:4840 , piece_id=JWOETSFSZ3HELLZNLEASYNGW5OR64TMQVOFYYJAJVTHWCBHVTSLQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: read tcp 188.166.225.170:39946->77.90.31.18:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231017812 (263h19m32.662097943s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 188.166.225.170:44164->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231063489 (263h19m32.66214362s) [noise=true , address=45.91.116.14:4203 , piece_id=WRLBES3FI5EA5AQLZXJYGDROP3KBG3E2F22R4YD6A5KKI44PLV2Q, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 188.166.225.170:51738->45.91.116.14:4203: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231090952 (263h19m32.662171083s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: read tcp 188.166.225.170:57158->77.90.31.15:4707: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231110719 (263h19m32.66219085s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:50784->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231140165 (263h19m32.662220296s) [noise=true , address=77.90.31.16:4749 , piece_id=XCPXVBYLQH345FPIW4KDNLBFUCDDEW2H4B25WIXGOO5FVMAIVKPQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: read tcp 188.166.225.170:51668->77.90.31.16:4749: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231144814 (263h19m32.662224945s) [noise=true , address=77.90.31.19:4865 , piece_id=B3HUFIJZKVZJH4OBBO2DD7S3I7HX4LO2IQ74XJPO4DTW4QER35NA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1bguBghQonV5sTHaWteyUEnpp5KNzBKbvBpbvrUNFDCveHN77X, address:77.90.31.19:4865): protocol: read tcp 188.166.225.170:39398->77.90.31.19:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231177167 (263h19m32.662257298s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: read tcp 188.166.225.170:58526->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231184571 (263h19m32.662264702s) [noise=true , address=45.91.116.12:4236 , piece_id=LCRQS2GOSHOSLZBGQTGB3Q3PA4VN7KXNW66TDZ5O424TPJWTQE3A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH, address:45.91.116.12:4236): protocol: read tcp 188.166.225.170:57166->45.91.116.12:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231189770 (263h19m32.662269901s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 188.166.225.170:54242->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231215850 (263h19m32.662295981s) [seg_pos={0 0}, upload=273764]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 19 16:45:38.459100173 (263h19m32.890180304s) [upload=273764]: Done uploading segments... Aug 19 16:45:38.459128316 (263h19m32.890208447s) [upload=273764]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.61770742s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X....XX..............................] duration: 5.811861248s FAIL (2025-08-19 17:11:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 19 16:49:12.801154313 (263h23m7.232234445s) [upload=273839]: Starting upload Aug 19 16:49:12.801215942 (263h23m7.232296073s) [upload=273839]: Done starting upload Aug 19 16:49:12.801330351 (263h23m7.232410482s) [upload=273839]: Uploading segments... Aug 19 16:49:12.801425372 (263h23m7.232505503s) [upload=273839]: Got next segment. Inline: false Aug 19 16:49:12.801455880 (263h23m7.232536012s) [upload=273839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:49:12.801461982 (263h23m7.232542113s) [upload=273839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:49:13.036708228 (263h23m7.467788359s) [seg_pos={0 0}, upload=273839]: Begin upload segment... Aug 19 16:49:13.036749197 (263h23m7.467829328s) [seg_pos={0 0}, upload=273839]: Done begin upload segment. Aug 19 16:49:13.036771971 (263h23m7.467852092s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.036780256 (263h23m7.467860437s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037135917 (263h23m7.468216048s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037326592 (263h23m7.468406723s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037403259 (263h23m7.46848339s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037667814 (263h23m7.468747945s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037761764 (263h23m7.468841896s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037943331 (263h23m7.469023462s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038070565 (263h23m7.469150696s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038168622 (263h23m7.469248753s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038300104 (263h23m7.469380235s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038396789 (263h23m7.46947692s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038622241 (263h23m7.469702362s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038679350 (263h23m7.469759481s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038854735 (263h23m7.469934856s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.039048045 (263h23m7.470128166s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:14.749064365 (263h23m9.180144496s) [upload=273839]: Next returned nil segment Aug 19 16:49:14.749117217 (263h23m9.180197348s) [upload=273839]: Waiting for error group managing segments... Aug 19 16:49:15.090317863 (263h23m9.521397994s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.132109007 (263h23m9.563189138s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.222090156 (263h23m9.653170287s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.328041780 (263h23m9.759121911s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.353998781 (263h23m9.785078912s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:36.709543777 (263h23m31.140623898s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:37.041062020 (263h23m31.472142141s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.183707791 (263h23m32.614787922s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.694075084 (263h23m33.125155215s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.716788163 (263h23m34.147868294s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.851890717 (263h23m34.282970848s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:40.293306092 (263h23m34.724386243s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:42.156929229 (263h23m36.58800936s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157075889 (263h23m36.58815601s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157126075 (263h23m36.588206206s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection Aug 19 16:49:42.157170951 (263h23m36.588251082s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: read tcp 188.166.225.170:34242->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:42.157215747 (263h23m36.588295878s) [seg_pos={0 0}, upload=273839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 19 16:49:42.385478364 (263h23m36.816558495s) [upload=273839]: Done uploading segments... Aug 19 16:49:42.385503091 (263h23m36.816583212s) [upload=273839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.811886066s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................X...X................................] duration: 6.46765231s FAIL (2025-08-19 17:11:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.4676732s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [......X..........................................X..........] duration: 2.053676828s FAIL (2025-08-19 17:11:14 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 17:04:06.343826144 (263h38m0.774906265s) [upload=274088]: Starting upload Aug 19 17:04:06.343875278 (263h38m0.774955409s) [upload=274088]: Done starting upload Aug 19 17:04:06.343971703 (263h38m0.775051834s) [upload=274088]: Uploading segments... Aug 19 17:04:06.344028321 (263h38m0.775108453s) [upload=274088]: Got next segment. Inline: true Aug 19 17:04:06.344033120 (263h38m0.775113252s) [upload=274088]: Waiting for error group managing segments... Aug 19 17:04:06.344068187 (263h38m0.775148318s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:04:06.344071964 (263h38m0.775152095s) [upload=274088]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:04:06.344074499 (263h38m0.77515462s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:04:06.653287654 (263h38m1.084367785s) [upload=274088]: Done uploading segments... Aug 19 17:04:06.653300519 (263h38m1.08438065s) [upload=274088]: Upload finished. err: <nil> Duration:* 2.05371924s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X....X.............................] duration: 6.175927732s FAIL (2025-08-19 17:11:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.175955714s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.812925213s FAIL (2025-08-19 17:11:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 263h18m33.75518017s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324344006 (263h18m33.755424127s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324633220 (263h18m33.755713351s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324851286 (263h18m33.755931417s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325068942 (263h18m33.756149063s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325283613 (263h18m33.756363744s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325499857 (263h18m33.756579998s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325759794 (263h18m33.756839915s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326059428 (263h18m33.757139549s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326277565 (263h18m33.757357696s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326470835 (263h18m33.757550966s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326673753 (263h18m33.757753884s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326867664 (263h18m33.757947795s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:40.851885389 (263h18m35.28296552s) [upload=273764]: Next returned nil segment Aug 19 16:44:40.851903334 (263h18m35.282983465s) [upload=273764]: Waiting for error group managing segments... Aug 19 16:44:41.244668372 (263h18m35.675748503s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.277280365 (263h18m35.708360496s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.328561665 (263h18m35.759641786s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.331747139 (263h18m35.76282727s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.357720320 (263h18m35.788800481s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.378190805 (263h18m35.809270936s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.486480333 (263h18m35.917560464s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:45:38.230570787 (263h19m32.661650918s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: expected piece hash; context deadline exceeded; EOF Aug 19 16:45:38.230979869 (263h19m32.66206s) [noise=true , address=77.90.31.18:4840 , piece_id=JWOETSFSZ3HELLZNLEASYNGW5OR64TMQVOFYYJAJVTHWCBHVTSLQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: read tcp 188.166.225.170:39946->77.90.31.18:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231017812 (263h19m32.662097943s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 188.166.225.170:44164->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231063489 (263h19m32.66214362s) [noise=true , address=45.91.116.14:4203 , piece_id=WRLBES3FI5EA5AQLZXJYGDROP3KBG3E2F22R4YD6A5KKI44PLV2Q, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 188.166.225.170:51738->45.91.116.14:4203: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231090952 (263h19m32.662171083s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: read tcp 188.166.225.170:57158->77.90.31.15:4707: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231110719 (263h19m32.66219085s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:50784->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231140165 (263h19m32.662220296s) [noise=true , address=77.90.31.16:4749 , piece_id=XCPXVBYLQH345FPIW4KDNLBFUCDDEW2H4B25WIXGOO5FVMAIVKPQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: read tcp 188.166.225.170:51668->77.90.31.16:4749: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231144814 (263h19m32.662224945s) [noise=true , address=77.90.31.19:4865 , piece_id=B3HUFIJZKVZJH4OBBO2DD7S3I7HX4LO2IQ74XJPO4DTW4QER35NA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1bguBghQonV5sTHaWteyUEnpp5KNzBKbvBpbvrUNFDCveHN77X, address:77.90.31.19:4865): protocol: read tcp 188.166.225.170:39398->77.90.31.19:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231177167 (263h19m32.662257298s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: read tcp 188.166.225.170:58526->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231184571 (263h19m32.662264702s) [noise=true , address=45.91.116.12:4236 , piece_id=LCRQS2GOSHOSLZBGQTGB3Q3PA4VN7KXNW66TDZ5O424TPJWTQE3A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH, address:45.91.116.12:4236): protocol: read tcp 188.166.225.170:57166->45.91.116.12:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231189770 (263h19m32.662269901s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 188.166.225.170:54242->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231215850 (263h19m32.662295981s) [seg_pos={0 0}, upload=273764]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 19 16:45:38.459100173 (263h19m32.890180304s) [upload=273764]: Done uploading segments... Aug 19 16:45:38.459128316 (263h19m32.890208447s) [upload=273764]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.812946042s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X...X..............X......X.XX........................X..] duration: 5.978611813s FAIL (2025-08-19 17:10:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== Aug 19 17:09:15.684096389 (263h43m10.11517652s) [upload=274176]: Starting upload Aug 19 17:09:15.684143008 (263h43m10.115223139s) [upload=274176]: Done starting upload Aug 19 17:09:15.684256185 (263h43m10.115336306s) [upload=274176]: Uploading segments... Aug 19 17:09:15.684468862 (263h43m10.115548993s) [upload=274176]: Got next segment. Inline: false Aug 19 17:09:15.684504950 (263h43m10.115585082s) [upload=274176]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:09:15.684509770 (263h43m10.115589901s) [upload=274176]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:09:16.135567491 (263h43m10.566647622s) [seg_pos={0 0}, upload=274176]: Begin upload segment... Aug 19 17:09:16.135613589 (263h43m10.56669372s) [seg_pos={0 0}, upload=274176]: Done begin upload segment. Aug 19 17:09:16.135636573 (263h43m10.566716695s) [noise=true , address=104.36.82.5:4853 , piece_id=K6TRSISY75NXYJNAAEALQ7W4RA26P5TGIVRMHX5SGOIF2GJ7X3GQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.135699574 (263h43m10.566779705s) [noise=true , address=104.36.82.60:4341 , piece_id=5NWVHAFJXPCBGPMRT42H4C3LPE3XKBDKPSX44D3DOWF6XXXJ5LFA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136028283 (263h43m10.567108404s) [noise=true , address=79.127.220.98:4432 , piece_id=4VA6KKCGA5GM55N3SD7FNUILD5XWYHE5VFRBFDIBAGVEK2VHQELA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136206604 (263h43m10.567286735s) [noise=true , address=104.36.82.22:4538 , piece_id=YUOOOPSZLQRNVU557APDO5FTOCY4FEYAHCBZQEE6YWMBPK5ZJYNA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136295745 (263h43m10.567375876s) [noise=true , address=77.90.31.11:4564 , piece_id=TX3C4VLEEOJOHBN7VDK4KIROG42PDDCB3SK26FZA7PM2455YZCMA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136640434 (263h43m10.567720565s) [noise=true , address=104.36.82.16:4315 , piece_id=P3SYICVJ5HSVY4MM62FDXQXJARZLIOQYNPBCT2SKYFPYZ3G3MSWQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136649702 (263h43m10.567729833s) [noise=true , address=104.36.82.44:4874 , piece_id=D3L2BM57MTGKIWKRSSEL7DRXUOK7LLKHL7M3Y3WAFCBILVDEV24Q, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136950888 (263h43m10.568031019s) [noise=true , address=104.36.82.35:4605 , piece_id=ORHH35ESP7KPGGDRD7ANECFHW7YTA2VFZTB5EMRCBQX5EWB7ZU4Q, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136961468 (263h43m10.568041599s) [noise=true , address=77.90.31.10:4535 , piece_id=N6LDVXERZAIR3EZ2TAMFQBM5Y565FMSP44I5KAWC32DTIOOM3FIQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137201027 (263h43m10.568281158s) [noise=true , address=77.90.31.12:4611 , piece_id=BANXKRHIMMOQFBSC4FW2E3SMNXCQERG2OTD7GXMM56T3X6JPNQGQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137260470 (263h43m10.568340601s) [noise=true , address=104.36.82.11:4134 , piece_id=BSGC4ADI6LMK3E2YEYPWFMYS3GC5VA42YM22VIJEGZ2JANYMLBYQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137463418 (263h43m10.568543549s) [noise=true , address=45.91.116.10:4110 , piece_id=KCZZOHPLSRNA45VXOMW7AKUYMRRFRUA52KSOBHNDRJPCB3RMV5HA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137553992 (263h43m10.568634123s) [noise=true , address=104.36.82.55:4224 , piece_id=PGZXUHGOONWDPIB4ZHAKLL4F2UWMZJPAL7BOTXIGSQWESLUPWGQA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137682447 (263h43m10.568762578s) [noise=true , address=45.91.116.12:4238 , piece_id=JAUKDCYIMCIOX4EFHQMCNCBLVZEIBKODPBL5QVBPOJIB554KLNRQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137883032 (263h43m10.568963163s) [noise=true , address=77.90.31.16:4743 , piece_id=YKPMLYPRMAZEBX7ORWEICH5AWKVS2J74D25W4ANIJJOUIBX5UBRA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137912418 (263h43m10.568992539s) [noise=true , address=104.36.82.23:4455 , piece_id=L2M5P5LRKVLVVDSYIOBV4SR3EAH2UO2JUZBGKZKIWBUIGFZPVEOA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.138136216 (263h43m10.569216347s) [noise=true , address=104.36.82.7:4930 , piece_id=EGTR52TZDAVTPPRCJLK4GOUXUWW2U7XJMQOV5P3JCKAKOK7GG5JQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.138374362 (263h43m10.569454493s) [noise=true , address=77.90.31.18:4839 , piece_id=BG553GCZSJEH5ORINSIQ7IUYU3UBQ3WO73ZAGRB7W45HOASJMBVQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:17.616571604 (263h43m12.047651765s) [upload=274176]: Next returned nil segment Aug 19 17:09:17.616591412 (263h43m12.047671543s) [upload=274176]: Waiting for error group managing segments... Aug 19 17:09:18.008239751 (263h43m12.439319882s) [noise=true , address=79.127.220.98:4432 , piece_id=4VA6KKCGA5GM55N3SD7FNUILD5XWYHE5VFRBFDIBAGVEK2VHQELA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.039911371 (263h43m12.470991502s) [noise=true , address=45.91.116.12:4238 , piece_id=JAUKDCYIMCIOX4EFHQMCNCBLVZEIBKODPBL5QVBPOJIB554KLNRQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.065695686 (263h43m12.496775817s) [noise=true , address=77.90.31.16:4743 , piece_id=YKPMLYPRMAZEBX7ORWEICH5AWKVS2J74D25W4ANIJJOUIBX5UBRA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.080085439 (263h43m12.51116557s) [noise=true , address=45.91.116.10:4110 , piece_id=KCZZOHPLSRNA45VXOMW7AKUYMRRFRUA52KSOBHNDRJPCB3RMV5HA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.131454157 (263h43m12.562534288s) [noise=true , address=104.36.82.11:4134 , piece_id=BSGC4ADI6LMK3E2YEYPWFMYS3GC5VA42YM22VIJEGZ2JANYMLBYQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.143557214 (263h43m12.574637345s) [noise=true , address=104.36.82.7:4930 , piece_id=EGTR52TZDAVTPPRCJLK4GOUXUWW2U7XJMQOV5P3JCKAKOK7GG5JQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.156135631 (263h43m12.587215762s) [noise=true , address=77.90.31.12:4611 , piece_id=BANXKRHIMMOQFBSC4FW2E3SMNXCQERG2OTD7GXMM56T3X6JPNQGQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.159345252 (263h43m12.590425373s) [noise=true , address=104.36.82.55:4224 , piece_id=PGZXUHGOONWDPIB4ZHAKLL4F2UWMZJPAL7BOTXIGSQWESLUPWGQA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.169106219 (263h43m12.60018635s) [noise=true , address=104.36.82.22:4538 , piece_id=YUOOOPSZLQRNVU557APDO5FTOCY4FEYAHCBZQEE6YWMBPK5ZJYNA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.188273593 (263h43m12.619353724s) [noise=true , address=104.36.82.35:4605 , piece_id=ORHH35ESP7KPGGDRD7ANECFHW7YTA2VFZTB5EMRCBQX5EWB7ZU4Q, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.204131846 (263h43m12.635211977s) [noise=true , address=104.36.82.16:4315 , piece_id=P3SYICVJ5HSVY4MM62FDXQXJARZLIOQYNPBCT2SKYFPYZ3G3MSWQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.217541965 (263h43m12.648622096s) [noise=true , address=104.36.82.60:4341 , piece_id=5NWVHAFJXPCBGPMRT42H4C3LPE3XKBDKPSX44D3DOWF6XXXJ5LFA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.222565827 (263h43m12.653645948s) [noise=true , address=104.36.82.23:4455 , piece_id=L2M5P5LRKVLVVDSYIOBV4SR3EAH2UO2JUZBGKZKIWBUIGFZPVEOA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.377619011 (263h43m12.808699132s) [noise=true , address=104.36.82.44:4874 , piece_id=D3L2BM57MTGKIWKRSSEL7DRXUOK7LLKHL7M3Y3WAFCBILVDEV24Q, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.377646604 (263h43m12.808726725s) [seg_pos={0 0}, upload=274176]: Segment reached optimal threshold of 14 pieces. Aug 19 17:09:18.377754100 (263h43m12.808834231s) [noise=true , address=104.36.82.5:4853 , piece_id=K6TRSISY75NXYJNAAEALQ7W4RA26P5TGIVRMHX5SGOIF2GJ7X3GQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377895570 (263h43m12.808975701s) [noise=true , address=77.90.31.18:4839 , piece_id=BG553GCZSJEH5ORINSIQ7IUYU3UBQ3WO73ZAGRB7W45HOASJMBVQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gmWwyaQ6zM6QdayBFpXdE9VSE1aaQcBHFcukJoASjjHWCDsNx): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377915529 (263h43m12.80899566s) [noise=true , address=77.90.31.11:4564 , piece_id=TX3C4VLEEOJOHBN7VDK4KIROG42PDDCB3SK26FZA7PM2455YZCMA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377974832 (263h43m12.809054963s) [noise=true , address=77.90.31.10:4535 , piece_id=N6LDVXERZAIR3EZ2TAMFQBM5Y565FMSP44I5KAWC32DTIOOM3FIQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377999008 (263h43m12.80907914s) [seg_pos={0 0}, upload=274176]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 17:09:18.378145599 (263h43m12.80922573s) [upload=274176]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 17:09:18.378150488 (263h43m12.809230619s) [upload=274176]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:09:18.983488776 (263h43m13.414568907s) [upload=274176]: Done uploading segments... Aug 19 17:09:18.983507222 (263h43m13.414587353s) [upload=274176]: Upload finished. err: <nil> Duration: 5.978629869s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................X....XX.............................] duration: 5.832658849s FAIL (2025-08-19 17:10:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 19 16:49:12.801154313 (263h23m7.232234445s) [upload=273839]: Starting upload Aug 19 16:49:12.801215942 (263h23m7.232296073s) [upload=273839]: Done starting upload Aug 19 16:49:12.801330351 (263h23m7.232410482s) [upload=273839]: Uploading segments... Aug 19 16:49:12.801425372 (263h23m7.232505503s) [upload=273839]: Got next segment. Inline: false Aug 19 16:49:12.801455880 (263h23m7.232536012s) [upload=273839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:49:12.801461982 (263h23m7.232542113s) [upload=273839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:49:13.036708228 (263h23m7.467788359s) [seg_pos={0 0}, upload=273839]: Begin upload segment... Aug 19 16:49:13.036749197 (263h23m7.467829328s) [seg_pos={0 0}, upload=273839]: Done begin upload segment. Aug 19 16:49:13.036771971 (263h23m7.467852092s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.036780256 (263h23m7.467860437s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037135917 (263h23m7.468216048s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037326592 (263h23m7.468406723s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037403259 (263h23m7.46848339s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037667814 (263h23m7.468747945s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037761764 (263h23m7.468841896s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037943331 (263h23m7.469023462s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038070565 (263h23m7.469150696s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038168622 (263h23m7.469248753s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038300104 (263h23m7.469380235s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038396789 (263h23m7.46947692s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038622241 (263h23m7.469702362s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038679350 (263h23m7.469759481s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038854735 (263h23m7.469934856s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.039048045 (263h23m7.470128166s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:14.749064365 (263h23m9.180144496s) [upload=273839]: Next returned nil segment Aug 19 16:49:14.749117217 (263h23m9.180197348s) [upload=273839]: Waiting for error group managing segments... Aug 19 16:49:15.090317863 (263h23m9.521397994s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.132109007 (263h23m9.563189138s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.222090156 (263h23m9.653170287s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.328041780 (263h23m9.759121911s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.353998781 (263h23m9.785078912s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:36.709543777 (263h23m31.140623898s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:37.041062020 (263h23m31.472142141s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.183707791 (263h23m32.614787922s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.694075084 (263h23m33.125155215s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.716788163 (263h23m34.147868294s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.851890717 (263h23m34.282970848s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:40.293306092 (263h23m34.724386243s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:42.156929229 (263h23m36.58800936s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157075889 (263h23m36.58815601s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157126075 (263h23m36.588206206s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection Aug 19 16:49:42.157170951 (263h23m36.588251082s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: read tcp 188.166.225.170:34242->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:42.157215747 (263h23m36.588295878s) [seg_pos={0 0}, upload=273839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 19 16:49:42.385478364 (263h23m36.816558495s) [upload=273839]: Done uploading segments... Aug 19 16:49:42.385503091 (263h23m36.816583212s) [upload=273839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.832706931s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........................X...X...............................] duration: 6.485809532s FAIL (2025-08-19 17:10:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.485833749s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......X..........................................X.........] duration: 1.989227912s FAIL (2025-08-19 17:10:22 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 17:04:06.343826144 (263h38m0.774906265s) [upload=274088]: Starting upload Aug 19 17:04:06.343875278 (263h38m0.774955409s) [upload=274088]: Done starting upload Aug 19 17:04:06.343971703 (263h38m0.775051834s) [upload=274088]: Uploading segments... Aug 19 17:04:06.344028321 (263h38m0.775108453s) [upload=274088]: Got next segment. Inline: true Aug 19 17:04:06.344033120 (263h38m0.775113252s) [upload=274088]: Waiting for error group managing segments... Aug 19 17:04:06.344068187 (263h38m0.775148318s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:04:06.344071964 (263h38m0.775152095s) [upload=274088]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:04:06.344074499 (263h38m0.77515462s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:04:06.653287654 (263h38m1.084367785s) [upload=274088]: Done uploading segments... Aug 19 17:04:06.653300519 (263h38m1.08438065s) [upload=274088]: Upload finished. err: <nil> Duration:* 1.989263759s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [X...X...X..............X......X.XX........................X.] duration: 5.959941196s FAIL (2025-08-19 17:10:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== Aug 19 17:09:15.684096389 (263h43m10.11517652s) [upload=274176]: Starting upload Aug 19 17:09:15.684143008 (263h43m10.115223139s) [upload=274176]: Done starting upload Aug 19 17:09:15.684256185 (263h43m10.115336306s) [upload=274176]: Uploading segments... Aug 19 17:09:15.684468862 (263h43m10.115548993s) [upload=274176]: Got next segment. Inline: false Aug 19 17:09:15.684504950 (263h43m10.115585082s) [upload=274176]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:09:15.684509770 (263h43m10.115589901s) [upload=274176]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:09:16.135567491 (263h43m10.566647622s) [seg_pos={0 0}, upload=274176]: Begin upload segment... Aug 19 17:09:16.135613589 (263h43m10.56669372s) [seg_pos={0 0}, upload=274176]: Done begin upload segment. Aug 19 17:09:16.135636573 (263h43m10.566716695s) [noise=true , address=104.36.82.5:4853 , piece_id=K6TRSISY75NXYJNAAEALQ7W4RA26P5TGIVRMHX5SGOIF2GJ7X3GQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.135699574 (263h43m10.566779705s) [noise=true , address=104.36.82.60:4341 , piece_id=5NWVHAFJXPCBGPMRT42H4C3LPE3XKBDKPSX44D3DOWF6XXXJ5LFA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136028283 (263h43m10.567108404s) [noise=true , address=79.127.220.98:4432 , piece_id=4VA6KKCGA5GM55N3SD7FNUILD5XWYHE5VFRBFDIBAGVEK2VHQELA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136206604 (263h43m10.567286735s) [noise=true , address=104.36.82.22:4538 , piece_id=YUOOOPSZLQRNVU557APDO5FTOCY4FEYAHCBZQEE6YWMBPK5ZJYNA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136295745 (263h43m10.567375876s) [noise=true , address=77.90.31.11:4564 , piece_id=TX3C4VLEEOJOHBN7VDK4KIROG42PDDCB3SK26FZA7PM2455YZCMA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136640434 (263h43m10.567720565s) [noise=true , address=104.36.82.16:4315 , piece_id=P3SYICVJ5HSVY4MM62FDXQXJARZLIOQYNPBCT2SKYFPYZ3G3MSWQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136649702 (263h43m10.567729833s) [noise=true , address=104.36.82.44:4874 , piece_id=D3L2BM57MTGKIWKRSSEL7DRXUOK7LLKHL7M3Y3WAFCBILVDEV24Q, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136950888 (263h43m10.568031019s) [noise=true , address=104.36.82.35:4605 , piece_id=ORHH35ESP7KPGGDRD7ANECFHW7YTA2VFZTB5EMRCBQX5EWB7ZU4Q, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136961468 (263h43m10.568041599s) [noise=true , address=77.90.31.10:4535 , piece_id=N6LDVXERZAIR3EZ2TAMFQBM5Y565FMSP44I5KAWC32DTIOOM3FIQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137201027 (263h43m10.568281158s) [noise=true , address=77.90.31.12:4611 , piece_id=BANXKRHIMMOQFBSC4FW2E3SMNXCQERG2OTD7GXMM56T3X6JPNQGQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137260470 (263h43m10.568340601s) [noise=true , address=104.36.82.11:4134 , piece_id=BSGC4ADI6LMK3E2YEYPWFMYS3GC5VA42YM22VIJEGZ2JANYMLBYQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137463418 (263h43m10.568543549s) [noise=true , address=45.91.116.10:4110 , piece_id=KCZZOHPLSRNA45VXOMW7AKUYMRRFRUA52KSOBHNDRJPCB3RMV5HA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137553992 (263h43m10.568634123s) [noise=true , address=104.36.82.55:4224 , piece_id=PGZXUHGOONWDPIB4ZHAKLL4F2UWMZJPAL7BOTXIGSQWESLUPWGQA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137682447 (263h43m10.568762578s) [noise=true , address=45.91.116.12:4238 , piece_id=JAUKDCYIMCIOX4EFHQMCNCBLVZEIBKODPBL5QVBPOJIB554KLNRQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137883032 (263h43m10.568963163s) [noise=true , address=77.90.31.16:4743 , piece_id=YKPMLYPRMAZEBX7ORWEICH5AWKVS2J74D25W4ANIJJOUIBX5UBRA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137912418 (263h43m10.568992539s) [noise=true , address=104.36.82.23:4455 , piece_id=L2M5P5LRKVLVVDSYIOBV4SR3EAH2UO2JUZBGKZKIWBUIGFZPVEOA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.138136216 (263h43m10.569216347s) [noise=true , address=104.36.82.7:4930 , piece_id=EGTR52TZDAVTPPRCJLK4GOUXUWW2U7XJMQOV5P3JCKAKOK7GG5JQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.138374362 (263h43m10.569454493s) [noise=true , address=77.90.31.18:4839 , piece_id=BG553GCZSJEH5ORINSIQ7IUYU3UBQ3WO73ZAGRB7W45HOASJMBVQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:17.616571604 (263h43m12.047651765s) [upload=274176]: Next returned nil segment Aug 19 17:09:17.616591412 (263h43m12.047671543s) [upload=274176]: Waiting for error group managing segments... Aug 19 17:09:18.008239751 (263h43m12.439319882s) [noise=true , address=79.127.220.98:4432 , piece_id=4VA6KKCGA5GM55N3SD7FNUILD5XWYHE5VFRBFDIBAGVEK2VHQELA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.039911371 (263h43m12.470991502s) [noise=true , address=45.91.116.12:4238 , piece_id=JAUKDCYIMCIOX4EFHQMCNCBLVZEIBKODPBL5QVBPOJIB554KLNRQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.065695686 (263h43m12.496775817s) [noise=true , address=77.90.31.16:4743 , piece_id=YKPMLYPRMAZEBX7ORWEICH5AWKVS2J74D25W4ANIJJOUIBX5UBRA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.080085439 (263h43m12.51116557s) [noise=true , address=45.91.116.10:4110 , piece_id=KCZZOHPLSRNA45VXOMW7AKUYMRRFRUA52KSOBHNDRJPCB3RMV5HA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.131454157 (263h43m12.562534288s) [noise=true , address=104.36.82.11:4134 , piece_id=BSGC4ADI6LMK3E2YEYPWFMYS3GC5VA42YM22VIJEGZ2JANYMLBYQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.143557214 (263h43m12.574637345s) [noise=true , address=104.36.82.7:4930 , piece_id=EGTR52TZDAVTPPRCJLK4GOUXUWW2U7XJMQOV5P3JCKAKOK7GG5JQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.156135631 (263h43m12.587215762s) [noise=true , address=77.90.31.12:4611 , piece_id=BANXKRHIMMOQFBSC4FW2E3SMNXCQERG2OTD7GXMM56T3X6JPNQGQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.159345252 (263h43m12.590425373s) [noise=true , address=104.36.82.55:4224 , piece_id=PGZXUHGOONWDPIB4ZHAKLL4F2UWMZJPAL7BOTXIGSQWESLUPWGQA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.169106219 (263h43m12.60018635s) [noise=true , address=104.36.82.22:4538 , piece_id=YUOOOPSZLQRNVU557APDO5FTOCY4FEYAHCBZQEE6YWMBPK5ZJYNA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.188273593 (263h43m12.619353724s) [noise=true , address=104.36.82.35:4605 , piece_id=ORHH35ESP7KPGGDRD7ANECFHW7YTA2VFZTB5EMRCBQX5EWB7ZU4Q, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.204131846 (263h43m12.635211977s) [noise=true , address=104.36.82.16:4315 , piece_id=P3SYICVJ5HSVY4MM62FDXQXJARZLIOQYNPBCT2SKYFPYZ3G3MSWQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.217541965 (263h43m12.648622096s) [noise=true , address=104.36.82.60:4341 , piece_id=5NWVHAFJXPCBGPMRT42H4C3LPE3XKBDKPSX44D3DOWF6XXXJ5LFA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.222565827 (263h43m12.653645948s) [noise=true , address=104.36.82.23:4455 , piece_id=L2M5P5LRKVLVVDSYIOBV4SR3EAH2UO2JUZBGKZKIWBUIGFZPVEOA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.377619011 (263h43m12.808699132s) [noise=true , address=104.36.82.44:4874 , piece_id=D3L2BM57MTGKIWKRSSEL7DRXUOK7LLKHL7M3Y3WAFCBILVDEV24Q, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.377646604 (263h43m12.808726725s) [seg_pos={0 0}, upload=274176]: Segment reached optimal threshold of 14 pieces. Aug 19 17:09:18.377754100 (263h43m12.808834231s) [noise=true , address=104.36.82.5:4853 , piece_id=K6TRSISY75NXYJNAAEALQ7W4RA26P5TGIVRMHX5SGOIF2GJ7X3GQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377895570 (263h43m12.808975701s) [noise=true , address=77.90.31.18:4839 , piece_id=BG553GCZSJEH5ORINSIQ7IUYU3UBQ3WO73ZAGRB7W45HOASJMBVQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gmWwyaQ6zM6QdayBFpXdE9VSE1aaQcBHFcukJoASjjHWCDsNx): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377915529 (263h43m12.80899566s) [noise=true , address=77.90.31.11:4564 , piece_id=TX3C4VLEEOJOHBN7VDK4KIROG42PDDCB3SK26FZA7PM2455YZCMA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377974832 (263h43m12.809054963s) [noise=true , address=77.90.31.10:4535 , piece_id=N6LDVXERZAIR3EZ2TAMFQBM5Y565FMSP44I5KAWC32DTIOOM3FIQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377999008 (263h43m12.80907914s) [seg_pos={0 0}, upload=274176]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 17:09:18.378145599 (263h43m12.80922573s) [upload=274176]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 17:09:18.378150488 (263h43m12.809230619s) [upload=274176]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:09:18.983488776 (263h43m13.414568907s) [upload=274176]: Done uploading segments... Aug 19 17:09:18.983507222 (263h43m13.414587353s) [upload=274176]: Upload finished. err: <nil> Duration: 5.959971785s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........................X...X..............................] duration: 6.341507131s FAIL (2025-08-19 17:10:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.341519204s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.701021192s FAIL (2025-08-19 17:10:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 263h18m33.75518017s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324344006 (263h18m33.755424127s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324633220 (263h18m33.755713351s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324851286 (263h18m33.755931417s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325068942 (263h18m33.756149063s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325283613 (263h18m33.756363744s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325499857 (263h18m33.756579998s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325759794 (263h18m33.756839915s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326059428 (263h18m33.757139549s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326277565 (263h18m33.757357696s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326470835 (263h18m33.757550966s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326673753 (263h18m33.757753884s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326867664 (263h18m33.757947795s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:40.851885389 (263h18m35.28296552s) [upload=273764]: Next returned nil segment Aug 19 16:44:40.851903334 (263h18m35.282983465s) [upload=273764]: Waiting for error group managing segments... Aug 19 16:44:41.244668372 (263h18m35.675748503s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.277280365 (263h18m35.708360496s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.328561665 (263h18m35.759641786s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.331747139 (263h18m35.76282727s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.357720320 (263h18m35.788800481s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.378190805 (263h18m35.809270936s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.486480333 (263h18m35.917560464s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:45:38.230570787 (263h19m32.661650918s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: expected piece hash; context deadline exceeded; EOF Aug 19 16:45:38.230979869 (263h19m32.66206s) [noise=true , address=77.90.31.18:4840 , piece_id=JWOETSFSZ3HELLZNLEASYNGW5OR64TMQVOFYYJAJVTHWCBHVTSLQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: read tcp 188.166.225.170:39946->77.90.31.18:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231017812 (263h19m32.662097943s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 188.166.225.170:44164->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231063489 (263h19m32.66214362s) [noise=true , address=45.91.116.14:4203 , piece_id=WRLBES3FI5EA5AQLZXJYGDROP3KBG3E2F22R4YD6A5KKI44PLV2Q, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 188.166.225.170:51738->45.91.116.14:4203: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231090952 (263h19m32.662171083s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: read tcp 188.166.225.170:57158->77.90.31.15:4707: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231110719 (263h19m32.66219085s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:50784->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231140165 (263h19m32.662220296s) [noise=true , address=77.90.31.16:4749 , piece_id=XCPXVBYLQH345FPIW4KDNLBFUCDDEW2H4B25WIXGOO5FVMAIVKPQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: read tcp 188.166.225.170:51668->77.90.31.16:4749: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231144814 (263h19m32.662224945s) [noise=true , address=77.90.31.19:4865 , piece_id=B3HUFIJZKVZJH4OBBO2DD7S3I7HX4LO2IQ74XJPO4DTW4QER35NA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1bguBghQonV5sTHaWteyUEnpp5KNzBKbvBpbvrUNFDCveHN77X, address:77.90.31.19:4865): protocol: read tcp 188.166.225.170:39398->77.90.31.19:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231177167 (263h19m32.662257298s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: read tcp 188.166.225.170:58526->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231184571 (263h19m32.662264702s) [noise=true , address=45.91.116.12:4236 , piece_id=LCRQS2GOSHOSLZBGQTGB3Q3PA4VN7KXNW66TDZ5O424TPJWTQE3A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH, address:45.91.116.12:4236): protocol: read tcp 188.166.225.170:57166->45.91.116.12:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231189770 (263h19m32.662269901s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 188.166.225.170:54242->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231215850 (263h19m32.662295981s) [seg_pos={0 0}, upload=273764]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 19 16:45:38.459100173 (263h19m32.890180304s) [upload=273764]: Done uploading segments... Aug 19 16:45:38.459128316 (263h19m32.890208447s) [upload=273764]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.701042163s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X....X............................] duration: 6.153711469s FAIL (2025-08-19 17:09:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.153735264s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [........X..........................................X........] duration: 2.031445522s FAIL (2025-08-19 17:09:54 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 17:04:06.343826144 (263h38m0.774906265s) [upload=274088]: Starting upload Aug 19 17:04:06.343875278 (263h38m0.774955409s) [upload=274088]: Done starting upload Aug 19 17:04:06.343971703 (263h38m0.775051834s) [upload=274088]: Uploading segments... Aug 19 17:04:06.344028321 (263h38m0.775108453s) [upload=274088]: Got next segment. Inline: true Aug 19 17:04:06.344033120 (263h38m0.775113252s) [upload=274088]: Waiting for error group managing segments... Aug 19 17:04:06.344068187 (263h38m0.775148318s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:04:06.344071964 (263h38m0.775152095s) [upload=274088]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:04:06.344074499 (263h38m0.77515462s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:04:06.653287654 (263h38m1.084367785s) [upload=274088]: Done uploading segments... Aug 19 17:04:06.653300519 (263h38m1.08438065s) [upload=274088]: Upload finished. err: <nil> Duration:* 2.03146987s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........X..........................................X.......] duration: 1.814752227s FAIL (2025-08-19 17:09:28 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 17:04:06.343826144 (263h38m0.774906265s) [upload=274088]: Starting upload Aug 19 17:04:06.343875278 (263h38m0.774955409s) [upload=274088]: Done starting upload Aug 19 17:04:06.343971703 (263h38m0.775051834s) [upload=274088]: Uploading segments... Aug 19 17:04:06.344028321 (263h38m0.775108453s) [upload=274088]: Got next segment. Inline: true Aug 19 17:04:06.344033120 (263h38m0.775113252s) [upload=274088]: Waiting for error group managing segments... Aug 19 17:04:06.344068187 (263h38m0.775148318s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:04:06.344071964 (263h38m0.775152095s) [upload=274088]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:04:06.344074499 (263h38m0.77515462s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:04:06.653287654 (263h38m1.084367785s) [upload=274088]: Done uploading segments... Aug 19 17:04:06.653300519 (263h38m1.08438065s) [upload=274088]: Upload finished. err: <nil> Duration:* 1.814789679s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................X....XX............................] duration: 5.730377119s FAIL (2025-08-19 17:09:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 19 16:49:12.801154313 (263h23m7.232234445s) [upload=273839]: Starting upload Aug 19 16:49:12.801215942 (263h23m7.232296073s) [upload=273839]: Done starting upload Aug 19 16:49:12.801330351 (263h23m7.232410482s) [upload=273839]: Uploading segments... Aug 19 16:49:12.801425372 (263h23m7.232505503s) [upload=273839]: Got next segment. Inline: false Aug 19 16:49:12.801455880 (263h23m7.232536012s) [upload=273839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:49:12.801461982 (263h23m7.232542113s) [upload=273839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:49:13.036708228 (263h23m7.467788359s) [seg_pos={0 0}, upload=273839]: Begin upload segment... Aug 19 16:49:13.036749197 (263h23m7.467829328s) [seg_pos={0 0}, upload=273839]: Done begin upload segment. Aug 19 16:49:13.036771971 (263h23m7.467852092s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.036780256 (263h23m7.467860437s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037135917 (263h23m7.468216048s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037326592 (263h23m7.468406723s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037403259 (263h23m7.46848339s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037667814 (263h23m7.468747945s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037761764 (263h23m7.468841896s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037943331 (263h23m7.469023462s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038070565 (263h23m7.469150696s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038168622 (263h23m7.469248753s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038300104 (263h23m7.469380235s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038396789 (263h23m7.46947692s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038622241 (263h23m7.469702362s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038679350 (263h23m7.469759481s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038854735 (263h23m7.469934856s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.039048045 (263h23m7.470128166s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:14.749064365 (263h23m9.180144496s) [upload=273839]: Next returned nil segment Aug 19 16:49:14.749117217 (263h23m9.180197348s) [upload=273839]: Waiting for error group managing segments... Aug 19 16:49:15.090317863 (263h23m9.521397994s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.132109007 (263h23m9.563189138s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.222090156 (263h23m9.653170287s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.328041780 (263h23m9.759121911s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.353998781 (263h23m9.785078912s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:36.709543777 (263h23m31.140623898s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:37.041062020 (263h23m31.472142141s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.183707791 (263h23m32.614787922s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.694075084 (263h23m33.125155215s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.716788163 (263h23m34.147868294s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.851890717 (263h23m34.282970848s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:40.293306092 (263h23m34.724386243s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:42.156929229 (263h23m36.58800936s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157075889 (263h23m36.58815601s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157126075 (263h23m36.588206206s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection Aug 19 16:49:42.157170951 (263h23m36.588251082s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: read tcp 188.166.225.170:34242->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:42.157215747 (263h23m36.588295878s) [seg_pos={0 0}, upload=273839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 19 16:49:42.385478364 (263h23m36.816558495s) [upload=273839]: Done uploading segments... Aug 19 16:49:42.385503091 (263h23m36.816583212s) [upload=273839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.730401596s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................X...X.............................] duration: 6.431776456s FAIL (2025-08-19 17:09:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.431796945s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X...X...X..............X......X.XX........................X] duration: 4.803885584s FAIL (2025-08-19 17:09:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== Aug 19 17:09:15.684096389 (263h43m10.11517652s) [upload=274176]: Starting upload Aug 19 17:09:15.684143008 (263h43m10.115223139s) [upload=274176]: Done starting upload Aug 19 17:09:15.684256185 (263h43m10.115336306s) [upload=274176]: Uploading segments... Aug 19 17:09:15.684468862 (263h43m10.115548993s) [upload=274176]: Got next segment. Inline: false Aug 19 17:09:15.684504950 (263h43m10.115585082s) [upload=274176]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:09:15.684509770 (263h43m10.115589901s) [upload=274176]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:09:16.135567491 (263h43m10.566647622s) [seg_pos={0 0}, upload=274176]: Begin upload segment... Aug 19 17:09:16.135613589 (263h43m10.56669372s) [seg_pos={0 0}, upload=274176]: Done begin upload segment. Aug 19 17:09:16.135636573 (263h43m10.566716695s) [noise=true , address=104.36.82.5:4853 , piece_id=K6TRSISY75NXYJNAAEALQ7W4RA26P5TGIVRMHX5SGOIF2GJ7X3GQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.135699574 (263h43m10.566779705s) [noise=true , address=104.36.82.60:4341 , piece_id=5NWVHAFJXPCBGPMRT42H4C3LPE3XKBDKPSX44D3DOWF6XXXJ5LFA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136028283 (263h43m10.567108404s) [noise=true , address=79.127.220.98:4432 , piece_id=4VA6KKCGA5GM55N3SD7FNUILD5XWYHE5VFRBFDIBAGVEK2VHQELA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136206604 (263h43m10.567286735s) [noise=true , address=104.36.82.22:4538 , piece_id=YUOOOPSZLQRNVU557APDO5FTOCY4FEYAHCBZQEE6YWMBPK5ZJYNA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136295745 (263h43m10.567375876s) [noise=true , address=77.90.31.11:4564 , piece_id=TX3C4VLEEOJOHBN7VDK4KIROG42PDDCB3SK26FZA7PM2455YZCMA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136640434 (263h43m10.567720565s) [noise=true , address=104.36.82.16:4315 , piece_id=P3SYICVJ5HSVY4MM62FDXQXJARZLIOQYNPBCT2SKYFPYZ3G3MSWQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136649702 (263h43m10.567729833s) [noise=true , address=104.36.82.44:4874 , piece_id=D3L2BM57MTGKIWKRSSEL7DRXUOK7LLKHL7M3Y3WAFCBILVDEV24Q, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136950888 (263h43m10.568031019s) [noise=true , address=104.36.82.35:4605 , piece_id=ORHH35ESP7KPGGDRD7ANECFHW7YTA2VFZTB5EMRCBQX5EWB7ZU4Q, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.136961468 (263h43m10.568041599s) [noise=true , address=77.90.31.10:4535 , piece_id=N6LDVXERZAIR3EZ2TAMFQBM5Y565FMSP44I5KAWC32DTIOOM3FIQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137201027 (263h43m10.568281158s) [noise=true , address=77.90.31.12:4611 , piece_id=BANXKRHIMMOQFBSC4FW2E3SMNXCQERG2OTD7GXMM56T3X6JPNQGQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137260470 (263h43m10.568340601s) [noise=true , address=104.36.82.11:4134 , piece_id=BSGC4ADI6LMK3E2YEYPWFMYS3GC5VA42YM22VIJEGZ2JANYMLBYQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137463418 (263h43m10.568543549s) [noise=true , address=45.91.116.10:4110 , piece_id=KCZZOHPLSRNA45VXOMW7AKUYMRRFRUA52KSOBHNDRJPCB3RMV5HA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137553992 (263h43m10.568634123s) [noise=true , address=104.36.82.55:4224 , piece_id=PGZXUHGOONWDPIB4ZHAKLL4F2UWMZJPAL7BOTXIGSQWESLUPWGQA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137682447 (263h43m10.568762578s) [noise=true , address=45.91.116.12:4238 , piece_id=JAUKDCYIMCIOX4EFHQMCNCBLVZEIBKODPBL5QVBPOJIB554KLNRQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137883032 (263h43m10.568963163s) [noise=true , address=77.90.31.16:4743 , piece_id=YKPMLYPRMAZEBX7ORWEICH5AWKVS2J74D25W4ANIJJOUIBX5UBRA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.137912418 (263h43m10.568992539s) [noise=true , address=104.36.82.23:4455 , piece_id=L2M5P5LRKVLVVDSYIOBV4SR3EAH2UO2JUZBGKZKIWBUIGFZPVEOA, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.138136216 (263h43m10.569216347s) [noise=true , address=104.36.82.7:4930 , piece_id=EGTR52TZDAVTPPRCJLK4GOUXUWW2U7XJMQOV5P3JCKAKOK7GG5JQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:16.138374362 (263h43m10.569454493s) [noise=true , address=77.90.31.18:4839 , piece_id=BG553GCZSJEH5ORINSIQ7IUYU3UBQ3WO73ZAGRB7W45HOASJMBVQ, seg_pos={0 0}, upload=274176]: Uploading piece... Aug 19 17:09:17.616571604 (263h43m12.047651765s) [upload=274176]: Next returned nil segment Aug 19 17:09:17.616591412 (263h43m12.047671543s) [upload=274176]: Waiting for error group managing segments... Aug 19 17:09:18.008239751 (263h43m12.439319882s) [noise=true , address=79.127.220.98:4432 , piece_id=4VA6KKCGA5GM55N3SD7FNUILD5XWYHE5VFRBFDIBAGVEK2VHQELA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.039911371 (263h43m12.470991502s) [noise=true , address=45.91.116.12:4238 , piece_id=JAUKDCYIMCIOX4EFHQMCNCBLVZEIBKODPBL5QVBPOJIB554KLNRQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.065695686 (263h43m12.496775817s) [noise=true , address=77.90.31.16:4743 , piece_id=YKPMLYPRMAZEBX7ORWEICH5AWKVS2J74D25W4ANIJJOUIBX5UBRA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.080085439 (263h43m12.51116557s) [noise=true , address=45.91.116.10:4110 , piece_id=KCZZOHPLSRNA45VXOMW7AKUYMRRFRUA52KSOBHNDRJPCB3RMV5HA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.131454157 (263h43m12.562534288s) [noise=true , address=104.36.82.11:4134 , piece_id=BSGC4ADI6LMK3E2YEYPWFMYS3GC5VA42YM22VIJEGZ2JANYMLBYQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.143557214 (263h43m12.574637345s) [noise=true , address=104.36.82.7:4930 , piece_id=EGTR52TZDAVTPPRCJLK4GOUXUWW2U7XJMQOV5P3JCKAKOK7GG5JQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.156135631 (263h43m12.587215762s) [noise=true , address=77.90.31.12:4611 , piece_id=BANXKRHIMMOQFBSC4FW2E3SMNXCQERG2OTD7GXMM56T3X6JPNQGQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.159345252 (263h43m12.590425373s) [noise=true , address=104.36.82.55:4224 , piece_id=PGZXUHGOONWDPIB4ZHAKLL4F2UWMZJPAL7BOTXIGSQWESLUPWGQA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.169106219 (263h43m12.60018635s) [noise=true , address=104.36.82.22:4538 , piece_id=YUOOOPSZLQRNVU557APDO5FTOCY4FEYAHCBZQEE6YWMBPK5ZJYNA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.188273593 (263h43m12.619353724s) [noise=true , address=104.36.82.35:4605 , piece_id=ORHH35ESP7KPGGDRD7ANECFHW7YTA2VFZTB5EMRCBQX5EWB7ZU4Q, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.204131846 (263h43m12.635211977s) [noise=true , address=104.36.82.16:4315 , piece_id=P3SYICVJ5HSVY4MM62FDXQXJARZLIOQYNPBCT2SKYFPYZ3G3MSWQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.217541965 (263h43m12.648622096s) [noise=true , address=104.36.82.60:4341 , piece_id=5NWVHAFJXPCBGPMRT42H4C3LPE3XKBDKPSX44D3DOWF6XXXJ5LFA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.222565827 (263h43m12.653645948s) [noise=true , address=104.36.82.23:4455 , piece_id=L2M5P5LRKVLVVDSYIOBV4SR3EAH2UO2JUZBGKZKIWBUIGFZPVEOA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.377619011 (263h43m12.808699132s) [noise=true , address=104.36.82.44:4874 , piece_id=D3L2BM57MTGKIWKRSSEL7DRXUOK7LLKHL7M3Y3WAFCBILVDEV24Q, seg_pos={0 0}, upload=274176]: Done uploading piece. err: <nil> Aug 19 17:09:18.377646604 (263h43m12.808726725s) [seg_pos={0 0}, upload=274176]: Segment reached optimal threshold of 14 pieces. Aug 19 17:09:18.377754100 (263h43m12.808834231s) [noise=true , address=104.36.82.5:4853 , piece_id=K6TRSISY75NXYJNAAEALQ7W4RA26P5TGIVRMHX5SGOIF2GJ7X3GQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377895570 (263h43m12.808975701s) [noise=true , address=77.90.31.18:4839 , piece_id=BG553GCZSJEH5ORINSIQ7IUYU3UBQ3WO73ZAGRB7W45HOASJMBVQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gmWwyaQ6zM6QdayBFpXdE9VSE1aaQcBHFcukJoASjjHWCDsNx): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377915529 (263h43m12.80899566s) [noise=true , address=77.90.31.11:4564 , piece_id=TX3C4VLEEOJOHBN7VDK4KIROG42PDDCB3SK26FZA7PM2455YZCMA, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377974832 (263h43m12.809054963s) [noise=true , address=77.90.31.10:4535 , piece_id=N6LDVXERZAIR3EZ2TAMFQBM5Y565FMSP44I5KAWC32DTIOOM3FIQ, seg_pos={0 0}, upload=274176]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz): protocol: expected piece hash; context canceled; EOF Aug 19 17:09:18.377999008 (263h43m12.80907914s) [seg_pos={0 0}, upload=274176]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 17:09:18.378145599 (263h43m12.80922573s) [upload=274176]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 17:09:18.378150488 (263h43m12.809230619s) [upload=274176]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:09:18.983488776 (263h43m13.414568907s) [upload=274176]: Done uploading segments... Aug 19 17:09:18.983507222 (263h43m13.414587353s) [upload=274176]: Upload finished. err: <nil> Duration: 4.803911483s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.69131805s FAIL (2025-08-19 17:09:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 263h18m33.75518017s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324344006 (263h18m33.755424127s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324633220 (263h18m33.755713351s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324851286 (263h18m33.755931417s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325068942 (263h18m33.756149063s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325283613 (263h18m33.756363744s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325499857 (263h18m33.756579998s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325759794 (263h18m33.756839915s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326059428 (263h18m33.757139549s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326277565 (263h18m33.757357696s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326470835 (263h18m33.757550966s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326673753 (263h18m33.757753884s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326867664 (263h18m33.757947795s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:40.851885389 (263h18m35.28296552s) [upload=273764]: Next returned nil segment Aug 19 16:44:40.851903334 (263h18m35.282983465s) [upload=273764]: Waiting for error group managing segments... Aug 19 16:44:41.244668372 (263h18m35.675748503s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.277280365 (263h18m35.708360496s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.328561665 (263h18m35.759641786s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.331747139 (263h18m35.76282727s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.357720320 (263h18m35.788800481s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.378190805 (263h18m35.809270936s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.486480333 (263h18m35.917560464s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:45:38.230570787 (263h19m32.661650918s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: expected piece hash; context deadline exceeded; EOF Aug 19 16:45:38.230979869 (263h19m32.66206s) [noise=true , address=77.90.31.18:4840 , piece_id=JWOETSFSZ3HELLZNLEASYNGW5OR64TMQVOFYYJAJVTHWCBHVTSLQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: read tcp 188.166.225.170:39946->77.90.31.18:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231017812 (263h19m32.662097943s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 188.166.225.170:44164->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231063489 (263h19m32.66214362s) [noise=true , address=45.91.116.14:4203 , piece_id=WRLBES3FI5EA5AQLZXJYGDROP3KBG3E2F22R4YD6A5KKI44PLV2Q, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 188.166.225.170:51738->45.91.116.14:4203: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231090952 (263h19m32.662171083s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: read tcp 188.166.225.170:57158->77.90.31.15:4707: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231110719 (263h19m32.66219085s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:50784->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231140165 (263h19m32.662220296s) [noise=true , address=77.90.31.16:4749 , piece_id=XCPXVBYLQH345FPIW4KDNLBFUCDDEW2H4B25WIXGOO5FVMAIVKPQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: read tcp 188.166.225.170:51668->77.90.31.16:4749: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231144814 (263h19m32.662224945s) [noise=true , address=77.90.31.19:4865 , piece_id=B3HUFIJZKVZJH4OBBO2DD7S3I7HX4LO2IQ74XJPO4DTW4QER35NA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1bguBghQonV5sTHaWteyUEnpp5KNzBKbvBpbvrUNFDCveHN77X, address:77.90.31.19:4865): protocol: read tcp 188.166.225.170:39398->77.90.31.19:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231177167 (263h19m32.662257298s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: read tcp 188.166.225.170:58526->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231184571 (263h19m32.662264702s) [noise=true , address=45.91.116.12:4236 , piece_id=LCRQS2GOSHOSLZBGQTGB3Q3PA4VN7KXNW66TDZ5O424TPJWTQE3A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH, address:45.91.116.12:4236): protocol: read tcp 188.166.225.170:57166->45.91.116.12:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231189770 (263h19m32.662269901s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 188.166.225.170:54242->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231215850 (263h19m32.662295981s) [seg_pos={0 0}, upload=273764]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 19 16:45:38.459100173 (263h19m32.890180304s) [upload=273764]: Done uploading segments... Aug 19 16:45:38.459128316 (263h19m32.890208447s) [upload=273764]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.691346717s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [X.X...X...X..............X......X.XX........................] duration: 5.832278021s FAIL (2025-08-19 17:09:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Aug 19 16:48:51.452491912 (263h22m45.883572043s) [upload=273834]: Starting upload Aug 19 16:48:51.452531028 (263h22m45.883611159s) [upload=273834]: Done starting upload Aug 19 16:48:51.452781006 (263h22m45.883861137s) [upload=273834]: Uploading segments... Aug 19 16:48:51.452853734 (263h22m45.883933866s) [upload=273834]: Got next segment. Inline: false Aug 19 16:48:51.452892670 (263h22m45.883972871s) [upload=273834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:48:51.452899813 (263h22m45.883980004s) [upload=273834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:48:51.962807465 (263h22m46.393887596s) [seg_pos={0 0}, upload=273834]: Begin upload segment... Aug 19 16:48:51.962852621 (263h22m46.393932743s) [seg_pos={0 0}, upload=273834]: Done begin upload segment. Aug 19 16:48:51.962875075 (263h22m46.393955206s) [noise=true , address=104.36.82.23:4470 , piece_id=5PDSSXU4HWGHBN62JJQBKHDQ6YYJVBWZZYSCJGKOPTZAZZUCAB6Q, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.962917746 (263h22m46.393997877s) [noise=true , address=66.23.193.175:4556 , piece_id=FBS7KZJF3HULYZZJDNLNOKMR5HSCR224IY7ZU4FUKZC5YMIIOAJA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963271022 (263h22m46.394351153s) [noise=true , address=45.91.116.9:4286 , piece_id=WYJJULBB3QVG2F46C52OCJWAMHVORGI7MDQS3CSYLF7REGAZQ3OQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963372556 (263h22m46.394452687s) [noise=true , address=156.146.43.225:4452 , piece_id=3T6HWTUPTX6A7BZULB4RSKGUO6GNDIHQGUP5W3EESJBWZGN7LLEQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963580684 (263h22m46.394660815s) [noise=true , address=104.36.82.35:4606 , piece_id=R37F22AIPFPV45K2PRWAJNONXQEDBJQPZGQKQ3LSHAQIBPKRXLHA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963653934 (263h22m46.394734066s) [noise=true , address=104.36.82.21:4478 , piece_id=XJWKXZ45KPRM3QUYXVIYU75M24GOCWGG5E3K4V62225XF3TGSS4Q, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963841874 (263h22m46.394922005s) [noise=true , address=79.127.201.215:4343 , piece_id=JF5HESY3ZARRZWZJHGTIG2BEZWWMXPHNJF5ALABQBAOVGJCHKI5A, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963910455 (263h22m46.394990586s) [noise=true , address=79.127.219.51:4211 , piece_id=OUB6QKVWCRJATMQFLAQ7UK3LJ4DILSLRAGFDP5YXROEXHZCPWGZA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964097293 (263h22m46.395177424s) [noise=true , address=104.36.82.60:4341 , piece_id=JCAHPCHE5IPHX35ZCDQ27I3DZ3R347HFRQWNBCNBXU6P3O7PTONA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964163640 (263h22m46.395243771s) [noise=true , address=77.90.31.18:4828 , piece_id=NFXT5LHIVX4G6DXMB4UWI37XCXLNHBRLVLXSJWTYAYN2JKCHRZQA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964400383 (263h22m46.395480514s) [noise=true , address=104.36.82.51:4052 , piece_id=7HLVSHJCDKI3TNKFETBUAY25HVIOPKAMSYQ6Q5BJ2NCJTQOTETTA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964624241 (263h22m46.395704372s) [noise=true , address=77.90.31.10:4529 , piece_id=HK5DLHGNULVF2RX7NNKFYN5JIFJB3LDEYXLXO3UN4D5OO4DJOW4A, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964832199 (263h22m46.39591233s) [noise=true , address=45.91.116.11:4130 , piece_id=5IZILPAZ4UKTFE766TWKT672JHDPWIQJI3TMOTASODK7NYG5BKPQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965048673 (263h22m46.396128804s) [noise=true , address=104.36.82.11:4127 , piece_id=6ACOOH24TTEMA4OQP6OFNHIABWDW3NDUVD44FSRIFLPTSVPHXOOA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965259626 (263h22m46.396339747s) [noise=true , address=77.90.31.15:4713 , piece_id=26S5ETN4OVNEL7IYYAIUBLP6UDS2QSSXCHD4H6YNHEQNY6FPOBOA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965452415 (263h22m46.396532546s) [noise=true , address=45.91.116.14:4209 , piece_id=5KAIVWUEWAG5NW6QGOAJLIMUADHZ6R4YTNAKTTXAGBRUEYXDAE7A, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965641637 (263h22m46.396721768s) [noise=true , address=45.91.116.7:4020 , piece_id=5NNTOPQ5BC7T2TLQ7NWSO3E6R4NDPB3JPXPMPRGG6I7YJU6QS4CQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965869794 (263h22m46.396949925s) [noise=true , address=77.90.31.17:4793 , piece_id=K3GCAOBIKBJV6Q3KOFXL5SEUFYE46MKJU5IAYNDA25IDCICTOFCQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:53.162668249 (263h22m47.59374838s) [upload=273834]: Next returned nil segment Aug 19 16:48:53.162683478 (263h22m47.593763599s) [upload=273834]: Waiting for error group managing segments... Aug 19 16:48:53.488132069 (263h22m47.91921223s) [noise=true , address=66.23.193.175:4556 , piece_id=FBS7KZJF3HULYZZJDNLNOKMR5HSCR224IY7ZU4FUKZC5YMIIOAJA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.887114541 (263h22m48.318194672s) [noise=true , address=79.127.219.51:4211 , piece_id=OUB6QKVWCRJATMQFLAQ7UK3LJ4DILSLRAGFDP5YXROEXHZCPWGZA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.925197211 (263h22m48.356277342s) [noise=true , address=156.146.43.225:4452 , piece_id=3T6HWTUPTX6A7BZULB4RSKGUO6GNDIHQGUP5W3EESJBWZGN7LLEQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.961937701 (263h22m48.393017832s) [noise=true , address=104.36.82.11:4127 , piece_id=6ACOOH24TTEMA4OQP6OFNHIABWDW3NDUVD44FSRIFLPTSVPHXOOA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.992310049 (263h22m48.42339018s) [noise=true , address=79.127.201.215:4343 , piece_id=JF5HESY3ZARRZWZJHGTIG2BEZWWMXPHNJF5ALABQBAOVGJCHKI5A, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.045243820 (263h22m48.476323961s) [noise=true , address=104.36.82.35:4606 , piece_id=R37F22AIPFPV45K2PRWAJNONXQEDBJQPZGQKQ3LSHAQIBPKRXLHA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.059875193 (263h22m48.490955324s) [noise=true , address=104.36.82.21:4478 , piece_id=XJWKXZ45KPRM3QUYXVIYU75M24GOCWGG5E3K4V62225XF3TGSS4Q, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.095629989 (263h22m48.52671012s) [noise=true , address=104.36.82.23:4470 , piece_id=5PDSSXU4HWGHBN62JJQBKHDQ6YYJVBWZZYSCJGKOPTZAZZUCAB6Q, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.103250568 (263h22m48.534330699s) [noise=true , address=104.36.82.51:4052 , piece_id=7HLVSHJCDKI3TNKFETBUAY25HVIOPKAMSYQ6Q5BJ2NCJTQOTETTA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.159963800 (263h22m48.591043921s) [noise=true , address=104.36.82.60:4341 , piece_id=JCAHPCHE5IPHX35ZCDQ27I3DZ3R347HFRQWNBCNBXU6P3O7PTONA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:49:37.214809078 (263h23m31.64588921s) [noise=true , address=77.90.31.17:4793 , piece_id=K3GCAOBIKBJV6Q3KOFXL5SEUFYE46MKJU5IAYNDA25IDCICTOFCQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:49:50.811468753 (263h23m45.242548894s) [noise=true , address=77.90.31.10:4529 , piece_id=HK5DLHGNULVF2RX7NNKFYN5JIFJB3LDEYXLXO3UN4D5OO4DJOW4A, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: read tcp 188.166.225.170:57426->77.90.31.10:4529: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811576489 (263h23m45.24265662s) [noise=true , address=45.91.116.14:4209 , piece_id=5KAIVWUEWAG5NW6QGOAJLIMUADHZ6R4YTNAKTTXAGBRUEYXDAE7A, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:12LWbAxsrDcyV31SxJNiVNYs7ERKAf6RTuKpE596UrYJAnXMndR, address:45.91.116.14:4209): protocol: rpc: dial tcp 45.91.116.14:4209: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811688974 (263h23m45.242769105s) [noise=true , address=45.91.116.9:4286 , piece_id=WYJJULBB3QVG2F46C52OCJWAMHVORGI7MDQS3CSYLF7REGAZQ3OQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:19DfbHig9tzF4VRHtSWkExwUS2bdZ8GFgKnAD9wvMBG7qaesNa, address:45.91.116.9:4286): protocol: read tcp 188.166.225.170:40032->45.91.116.9:4286: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811727057 (263h23m45.242807188s) [noise=true , address=45.91.116.11:4130 , piece_id=5IZILPAZ4UKTFE766TWKT672JHDPWIQJI3TMOTASODK7NYG5BKPQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:126ThYkfUAYf2JR8f5rkwyumcvmRArxKbpLUjhJukoc3FADHj2T, address:45.91.116.11:4130): protocol: read tcp 188.166.225.170:46860->45.91.116.11:4130: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811808934 (263h23m45.242889085s) [noise=true , address=45.91.116.7:4020 , piece_id=5NNTOPQ5BC7T2TLQ7NWSO3E6R4NDPB3JPXPMPRGG6I7YJU6QS4CQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:1MT5Nk9DRHLLfHi5sJnzWEjuL3aV2HnbS9Xebdw8vtBMjQmFnV, address:45.91.116.7:4020): protocol: read tcp 188.166.225.170:48966->45.91.116.7:4020: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811874559 (263h23m45.24295469s) [noise=true , address=77.90.31.18:4828 , piece_id=NFXT5LHIVX4G6DXMB4UWI37XCXLNHBRLVLXSJWTYAYN2JKCHRZQA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:1jwTDXPShXfNdfMx3UXi22gQphBpFre1MhRsHyjq6xRz4wwTr9, address:77.90.31.18:4828): protocol: read tcp 188.166.225.170:47612->77.90.31.18:4828: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811914556 (263h23m45.242994687s) [noise=true , address=77.90.31.15:4713 , piece_id=26S5ETN4OVNEL7IYYAIUBLP6UDS2QSSXCHD4H6YNHEQNY6FPOBOA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: read tcp 188.166.225.170:41506->77.90.31.15:4713: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811942720 (263h23m45.243022841s) [seg_pos={0 0}, upload=273834]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Aug 19 16:49:51.082762534 (263h23m45.513842665s) [upload=273834]: Done uploading segments... Aug 19 16:49:51.082792671 (263h23m45.513872803s) [upload=273834]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.832304894s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X....X...........................] duration: 6.022998803s FAIL (2025-08-19 17:08:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.023020776s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........X..........................................X......] duration: 1.845283066s FAIL (2025-08-19 17:08:39 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 17:04:06.343826144 (263h38m0.774906265s) [upload=274088]: Starting upload Aug 19 17:04:06.343875278 (263h38m0.774955409s) [upload=274088]: Done starting upload Aug 19 17:04:06.343971703 (263h38m0.775051834s) [upload=274088]: Uploading segments... Aug 19 17:04:06.344028321 (263h38m0.775108453s) [upload=274088]: Got next segment. Inline: true Aug 19 17:04:06.344033120 (263h38m0.775113252s) [upload=274088]: Waiting for error group managing segments... Aug 19 17:04:06.344068187 (263h38m0.775148318s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:04:06.344071964 (263h38m0.775152095s) [upload=274088]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:04:06.344074499 (263h38m0.77515462s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:04:06.653287654 (263h38m1.084367785s) [upload=274088]: Done uploading segments... Aug 19 17:04:06.653300519 (263h38m1.08438065s) [upload=274088]: Upload finished. err: <nil> Duration:* 1.845408535s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................X....XX...........................] duration: 5.862388178s FAIL (2025-08-19 17:08:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 19 16:49:12.801154313 (263h23m7.232234445s) [upload=273839]: Starting upload Aug 19 16:49:12.801215942 (263h23m7.232296073s) [upload=273839]: Done starting upload Aug 19 16:49:12.801330351 (263h23m7.232410482s) [upload=273839]: Uploading segments... Aug 19 16:49:12.801425372 (263h23m7.232505503s) [upload=273839]: Got next segment. Inline: false Aug 19 16:49:12.801455880 (263h23m7.232536012s) [upload=273839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:49:12.801461982 (263h23m7.232542113s) [upload=273839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:49:13.036708228 (263h23m7.467788359s) [seg_pos={0 0}, upload=273839]: Begin upload segment... Aug 19 16:49:13.036749197 (263h23m7.467829328s) [seg_pos={0 0}, upload=273839]: Done begin upload segment. Aug 19 16:49:13.036771971 (263h23m7.467852092s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.036780256 (263h23m7.467860437s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037135917 (263h23m7.468216048s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037326592 (263h23m7.468406723s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037403259 (263h23m7.46848339s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037667814 (263h23m7.468747945s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037761764 (263h23m7.468841896s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037943331 (263h23m7.469023462s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038070565 (263h23m7.469150696s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038168622 (263h23m7.469248753s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038300104 (263h23m7.469380235s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038396789 (263h23m7.46947692s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038622241 (263h23m7.469702362s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038679350 (263h23m7.469759481s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038854735 (263h23m7.469934856s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.039048045 (263h23m7.470128166s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:14.749064365 (263h23m9.180144496s) [upload=273839]: Next returned nil segment Aug 19 16:49:14.749117217 (263h23m9.180197348s) [upload=273839]: Waiting for error group managing segments... Aug 19 16:49:15.090317863 (263h23m9.521397994s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.132109007 (263h23m9.563189138s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.222090156 (263h23m9.653170287s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.328041780 (263h23m9.759121911s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.353998781 (263h23m9.785078912s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:36.709543777 (263h23m31.140623898s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:37.041062020 (263h23m31.472142141s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.183707791 (263h23m32.614787922s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.694075084 (263h23m33.125155215s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.716788163 (263h23m34.147868294s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.851890717 (263h23m34.282970848s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:40.293306092 (263h23m34.724386243s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:42.156929229 (263h23m36.58800936s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157075889 (263h23m36.58815601s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157126075 (263h23m36.588206206s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection Aug 19 16:49:42.157170951 (263h23m36.588251082s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: read tcp 188.166.225.170:34242->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:42.157215747 (263h23m36.588295878s) [seg_pos={0 0}, upload=273839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 19 16:49:42.385478364 (263h23m36.816558495s) [upload=273839]: Done uploading segments... Aug 19 16:49:42.385503091 (263h23m36.816583212s) [upload=273839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.86241451s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........................X...X............................] duration: 6.358381806s FAIL (2025-08-19 17:08:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.358404559s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X....X..........................] duration: 6.153352636s FAIL (2025-08-19 17:08:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.153382713s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X.X...X...X..............X......X.XX.......................] duration: 6.689186417s FAIL (2025-08-19 17:08:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Aug 19 16:48:51.452491912 (263h22m45.883572043s) [upload=273834]: Starting upload Aug 19 16:48:51.452531028 (263h22m45.883611159s) [upload=273834]: Done starting upload Aug 19 16:48:51.452781006 (263h22m45.883861137s) [upload=273834]: Uploading segments... Aug 19 16:48:51.452853734 (263h22m45.883933866s) [upload=273834]: Got next segment. Inline: false Aug 19 16:48:51.452892670 (263h22m45.883972871s) [upload=273834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:48:51.452899813 (263h22m45.883980004s) [upload=273834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:48:51.962807465 (263h22m46.393887596s) [seg_pos={0 0}, upload=273834]: Begin upload segment... Aug 19 16:48:51.962852621 (263h22m46.393932743s) [seg_pos={0 0}, upload=273834]: Done begin upload segment. Aug 19 16:48:51.962875075 (263h22m46.393955206s) [noise=true , address=104.36.82.23:4470 , piece_id=5PDSSXU4HWGHBN62JJQBKHDQ6YYJVBWZZYSCJGKOPTZAZZUCAB6Q, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.962917746 (263h22m46.393997877s) [noise=true , address=66.23.193.175:4556 , piece_id=FBS7KZJF3HULYZZJDNLNOKMR5HSCR224IY7ZU4FUKZC5YMIIOAJA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963271022 (263h22m46.394351153s) [noise=true , address=45.91.116.9:4286 , piece_id=WYJJULBB3QVG2F46C52OCJWAMHVORGI7MDQS3CSYLF7REGAZQ3OQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963372556 (263h22m46.394452687s) [noise=true , address=156.146.43.225:4452 , piece_id=3T6HWTUPTX6A7BZULB4RSKGUO6GNDIHQGUP5W3EESJBWZGN7LLEQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963580684 (263h22m46.394660815s) [noise=true , address=104.36.82.35:4606 , piece_id=R37F22AIPFPV45K2PRWAJNONXQEDBJQPZGQKQ3LSHAQIBPKRXLHA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963653934 (263h22m46.394734066s) [noise=true , address=104.36.82.21:4478 , piece_id=XJWKXZ45KPRM3QUYXVIYU75M24GOCWGG5E3K4V62225XF3TGSS4Q, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963841874 (263h22m46.394922005s) [noise=true , address=79.127.201.215:4343 , piece_id=JF5HESY3ZARRZWZJHGTIG2BEZWWMXPHNJF5ALABQBAOVGJCHKI5A, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963910455 (263h22m46.394990586s) [noise=true , address=79.127.219.51:4211 , piece_id=OUB6QKVWCRJATMQFLAQ7UK3LJ4DILSLRAGFDP5YXROEXHZCPWGZA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964097293 (263h22m46.395177424s) [noise=true , address=104.36.82.60:4341 , piece_id=JCAHPCHE5IPHX35ZCDQ27I3DZ3R347HFRQWNBCNBXU6P3O7PTONA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964163640 (263h22m46.395243771s) [noise=true , address=77.90.31.18:4828 , piece_id=NFXT5LHIVX4G6DXMB4UWI37XCXLNHBRLVLXSJWTYAYN2JKCHRZQA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964400383 (263h22m46.395480514s) [noise=true , address=104.36.82.51:4052 , piece_id=7HLVSHJCDKI3TNKFETBUAY25HVIOPKAMSYQ6Q5BJ2NCJTQOTETTA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964624241 (263h22m46.395704372s) [noise=true , address=77.90.31.10:4529 , piece_id=HK5DLHGNULVF2RX7NNKFYN5JIFJB3LDEYXLXO3UN4D5OO4DJOW4A, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964832199 (263h22m46.39591233s) [noise=true , address=45.91.116.11:4130 , piece_id=5IZILPAZ4UKTFE766TWKT672JHDPWIQJI3TMOTASODK7NYG5BKPQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965048673 (263h22m46.396128804s) [noise=true , address=104.36.82.11:4127 , piece_id=6ACOOH24TTEMA4OQP6OFNHIABWDW3NDUVD44FSRIFLPTSVPHXOOA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965259626 (263h22m46.396339747s) [noise=true , address=77.90.31.15:4713 , piece_id=26S5ETN4OVNEL7IYYAIUBLP6UDS2QSSXCHD4H6YNHEQNY6FPOBOA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965452415 (263h22m46.396532546s) [noise=true , address=45.91.116.14:4209 , piece_id=5KAIVWUEWAG5NW6QGOAJLIMUADHZ6R4YTNAKTTXAGBRUEYXDAE7A, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965641637 (263h22m46.396721768s) [noise=true , address=45.91.116.7:4020 , piece_id=5NNTOPQ5BC7T2TLQ7NWSO3E6R4NDPB3JPXPMPRGG6I7YJU6QS4CQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965869794 (263h22m46.396949925s) [noise=true , address=77.90.31.17:4793 , piece_id=K3GCAOBIKBJV6Q3KOFXL5SEUFYE46MKJU5IAYNDA25IDCICTOFCQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:53.162668249 (263h22m47.59374838s) [upload=273834]: Next returned nil segment Aug 19 16:48:53.162683478 (263h22m47.593763599s) [upload=273834]: Waiting for error group managing segments... Aug 19 16:48:53.488132069 (263h22m47.91921223s) [noise=true , address=66.23.193.175:4556 , piece_id=FBS7KZJF3HULYZZJDNLNOKMR5HSCR224IY7ZU4FUKZC5YMIIOAJA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.887114541 (263h22m48.318194672s) [noise=true , address=79.127.219.51:4211 , piece_id=OUB6QKVWCRJATMQFLAQ7UK3LJ4DILSLRAGFDP5YXROEXHZCPWGZA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.925197211 (263h22m48.356277342s) [noise=true , address=156.146.43.225:4452 , piece_id=3T6HWTUPTX6A7BZULB4RSKGUO6GNDIHQGUP5W3EESJBWZGN7LLEQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.961937701 (263h22m48.393017832s) [noise=true , address=104.36.82.11:4127 , piece_id=6ACOOH24TTEMA4OQP6OFNHIABWDW3NDUVD44FSRIFLPTSVPHXOOA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.992310049 (263h22m48.42339018s) [noise=true , address=79.127.201.215:4343 , piece_id=JF5HESY3ZARRZWZJHGTIG2BEZWWMXPHNJF5ALABQBAOVGJCHKI5A, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.045243820 (263h22m48.476323961s) [noise=true , address=104.36.82.35:4606 , piece_id=R37F22AIPFPV45K2PRWAJNONXQEDBJQPZGQKQ3LSHAQIBPKRXLHA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.059875193 (263h22m48.490955324s) [noise=true , address=104.36.82.21:4478 , piece_id=XJWKXZ45KPRM3QUYXVIYU75M24GOCWGG5E3K4V62225XF3TGSS4Q, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.095629989 (263h22m48.52671012s) [noise=true , address=104.36.82.23:4470 , piece_id=5PDSSXU4HWGHBN62JJQBKHDQ6YYJVBWZZYSCJGKOPTZAZZUCAB6Q, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.103250568 (263h22m48.534330699s) [noise=true , address=104.36.82.51:4052 , piece_id=7HLVSHJCDKI3TNKFETBUAY25HVIOPKAMSYQ6Q5BJ2NCJTQOTETTA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.159963800 (263h22m48.591043921s) [noise=true , address=104.36.82.60:4341 , piece_id=JCAHPCHE5IPHX35ZCDQ27I3DZ3R347HFRQWNBCNBXU6P3O7PTONA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:49:37.214809078 (263h23m31.64588921s) [noise=true , address=77.90.31.17:4793 , piece_id=K3GCAOBIKBJV6Q3KOFXL5SEUFYE46MKJU5IAYNDA25IDCICTOFCQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:49:50.811468753 (263h23m45.242548894s) [noise=true , address=77.90.31.10:4529 , piece_id=HK5DLHGNULVF2RX7NNKFYN5JIFJB3LDEYXLXO3UN4D5OO4DJOW4A, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: read tcp 188.166.225.170:57426->77.90.31.10:4529: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811576489 (263h23m45.24265662s) [noise=true , address=45.91.116.14:4209 , piece_id=5KAIVWUEWAG5NW6QGOAJLIMUADHZ6R4YTNAKTTXAGBRUEYXDAE7A, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:12LWbAxsrDcyV31SxJNiVNYs7ERKAf6RTuKpE596UrYJAnXMndR, address:45.91.116.14:4209): protocol: rpc: dial tcp 45.91.116.14:4209: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811688974 (263h23m45.242769105s) [noise=true , address=45.91.116.9:4286 , piece_id=WYJJULBB3QVG2F46C52OCJWAMHVORGI7MDQS3CSYLF7REGAZQ3OQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:19DfbHig9tzF4VRHtSWkExwUS2bdZ8GFgKnAD9wvMBG7qaesNa, address:45.91.116.9:4286): protocol: read tcp 188.166.225.170:40032->45.91.116.9:4286: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811727057 (263h23m45.242807188s) [noise=true , address=45.91.116.11:4130 , piece_id=5IZILPAZ4UKTFE766TWKT672JHDPWIQJI3TMOTASODK7NYG5BKPQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:126ThYkfUAYf2JR8f5rkwyumcvmRArxKbpLUjhJukoc3FADHj2T, address:45.91.116.11:4130): protocol: read tcp 188.166.225.170:46860->45.91.116.11:4130: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811808934 (263h23m45.242889085s) [noise=true , address=45.91.116.7:4020 , piece_id=5NNTOPQ5BC7T2TLQ7NWSO3E6R4NDPB3JPXPMPRGG6I7YJU6QS4CQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:1MT5Nk9DRHLLfHi5sJnzWEjuL3aV2HnbS9Xebdw8vtBMjQmFnV, address:45.91.116.7:4020): protocol: read tcp 188.166.225.170:48966->45.91.116.7:4020: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811874559 (263h23m45.24295469s) [noise=true , address=77.90.31.18:4828 , piece_id=NFXT5LHIVX4G6DXMB4UWI37XCXLNHBRLVLXSJWTYAYN2JKCHRZQA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:1jwTDXPShXfNdfMx3UXi22gQphBpFre1MhRsHyjq6xRz4wwTr9, address:77.90.31.18:4828): protocol: read tcp 188.166.225.170:47612->77.90.31.18:4828: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811914556 (263h23m45.242994687s) [noise=true , address=77.90.31.15:4713 , piece_id=26S5ETN4OVNEL7IYYAIUBLP6UDS2QSSXCHD4H6YNHEQNY6FPOBOA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: read tcp 188.166.225.170:41506->77.90.31.15:4713: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811942720 (263h23m45.243022841s) [seg_pos={0 0}, upload=273834]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Aug 19 16:49:51.082762534 (263h23m45.513842665s) [upload=273834]: Done uploading segments... Aug 19 16:49:51.082792671 (263h23m45.513872803s) [upload=273834]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 6.689204202s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.706670609s FAIL (2025-08-19 17:08:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 263h18m33.75518017s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324344006 (263h18m33.755424127s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324633220 (263h18m33.755713351s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324851286 (263h18m33.755931417s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325068942 (263h18m33.756149063s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325283613 (263h18m33.756363744s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325499857 (263h18m33.756579998s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325759794 (263h18m33.756839915s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326059428 (263h18m33.757139549s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326277565 (263h18m33.757357696s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326470835 (263h18m33.757550966s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326673753 (263h18m33.757753884s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326867664 (263h18m33.757947795s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:40.851885389 (263h18m35.28296552s) [upload=273764]: Next returned nil segment Aug 19 16:44:40.851903334 (263h18m35.282983465s) [upload=273764]: Waiting for error group managing segments... Aug 19 16:44:41.244668372 (263h18m35.675748503s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.277280365 (263h18m35.708360496s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.328561665 (263h18m35.759641786s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.331747139 (263h18m35.76282727s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.357720320 (263h18m35.788800481s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.378190805 (263h18m35.809270936s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.486480333 (263h18m35.917560464s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:45:38.230570787 (263h19m32.661650918s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: expected piece hash; context deadline exceeded; EOF Aug 19 16:45:38.230979869 (263h19m32.66206s) [noise=true , address=77.90.31.18:4840 , piece_id=JWOETSFSZ3HELLZNLEASYNGW5OR64TMQVOFYYJAJVTHWCBHVTSLQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: read tcp 188.166.225.170:39946->77.90.31.18:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231017812 (263h19m32.662097943s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 188.166.225.170:44164->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231063489 (263h19m32.66214362s) [noise=true , address=45.91.116.14:4203 , piece_id=WRLBES3FI5EA5AQLZXJYGDROP3KBG3E2F22R4YD6A5KKI44PLV2Q, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 188.166.225.170:51738->45.91.116.14:4203: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231090952 (263h19m32.662171083s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: read tcp 188.166.225.170:57158->77.90.31.15:4707: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231110719 (263h19m32.66219085s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:50784->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231140165 (263h19m32.662220296s) [noise=true , address=77.90.31.16:4749 , piece_id=XCPXVBYLQH345FPIW4KDNLBFUCDDEW2H4B25WIXGOO5FVMAIVKPQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: read tcp 188.166.225.170:51668->77.90.31.16:4749: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231144814 (263h19m32.662224945s) [noise=true , address=77.90.31.19:4865 , piece_id=B3HUFIJZKVZJH4OBBO2DD7S3I7HX4LO2IQ74XJPO4DTW4QER35NA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1bguBghQonV5sTHaWteyUEnpp5KNzBKbvBpbvrUNFDCveHN77X, address:77.90.31.19:4865): protocol: read tcp 188.166.225.170:39398->77.90.31.19:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231177167 (263h19m32.662257298s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: read tcp 188.166.225.170:58526->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231184571 (263h19m32.662264702s) [noise=true , address=45.91.116.12:4236 , piece_id=LCRQS2GOSHOSLZBGQTGB3Q3PA4VN7KXNW66TDZ5O424TPJWTQE3A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH, address:45.91.116.12:4236): protocol: read tcp 188.166.225.170:57166->45.91.116.12:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231189770 (263h19m32.662269901s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 188.166.225.170:54242->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231215850 (263h19m32.662295981s) [seg_pos={0 0}, upload=273764]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 19 16:45:38.459100173 (263h19m32.890180304s) [upload=273764]: Done uploading segments... Aug 19 16:45:38.459128316 (263h19m32.890208447s) [upload=273764]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.706701979s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................X....XX..........................] duration: 5.849253526s FAIL (2025-08-19 17:07:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 19 16:49:12.801154313 (263h23m7.232234445s) [upload=273839]: Starting upload Aug 19 16:49:12.801215942 (263h23m7.232296073s) [upload=273839]: Done starting upload Aug 19 16:49:12.801330351 (263h23m7.232410482s) [upload=273839]: Uploading segments... Aug 19 16:49:12.801425372 (263h23m7.232505503s) [upload=273839]: Got next segment. Inline: false Aug 19 16:49:12.801455880 (263h23m7.232536012s) [upload=273839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:49:12.801461982 (263h23m7.232542113s) [upload=273839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:49:13.036708228 (263h23m7.467788359s) [seg_pos={0 0}, upload=273839]: Begin upload segment... Aug 19 16:49:13.036749197 (263h23m7.467829328s) [seg_pos={0 0}, upload=273839]: Done begin upload segment. Aug 19 16:49:13.036771971 (263h23m7.467852092s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.036780256 (263h23m7.467860437s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037135917 (263h23m7.468216048s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037326592 (263h23m7.468406723s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037403259 (263h23m7.46848339s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037667814 (263h23m7.468747945s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037761764 (263h23m7.468841896s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037943331 (263h23m7.469023462s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038070565 (263h23m7.469150696s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038168622 (263h23m7.469248753s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038300104 (263h23m7.469380235s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038396789 (263h23m7.46947692s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038622241 (263h23m7.469702362s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038679350 (263h23m7.469759481s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038854735 (263h23m7.469934856s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.039048045 (263h23m7.470128166s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:14.749064365 (263h23m9.180144496s) [upload=273839]: Next returned nil segment Aug 19 16:49:14.749117217 (263h23m9.180197348s) [upload=273839]: Waiting for error group managing segments... Aug 19 16:49:15.090317863 (263h23m9.521397994s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.132109007 (263h23m9.563189138s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.222090156 (263h23m9.653170287s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.328041780 (263h23m9.759121911s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.353998781 (263h23m9.785078912s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:36.709543777 (263h23m31.140623898s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:37.041062020 (263h23m31.472142141s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.183707791 (263h23m32.614787922s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.694075084 (263h23m33.125155215s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.716788163 (263h23m34.147868294s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.851890717 (263h23m34.282970848s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:40.293306092 (263h23m34.724386243s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:42.156929229 (263h23m36.58800936s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157075889 (263h23m36.58815601s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157126075 (263h23m36.588206206s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection Aug 19 16:49:42.157170951 (263h23m36.588251082s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: read tcp 188.166.225.170:34242->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:42.157215747 (263h23m36.588295878s) [seg_pos={0 0}, upload=273839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 19 16:49:42.385478364 (263h23m36.816558495s) [upload=273839]: Done uploading segments... Aug 19 16:49:42.385503091 (263h23m36.816583212s) [upload=273839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.849289003s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............................X...X...........................] duration: 6.410496388s FAIL (2025-08-19 17:07:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.410514353s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...........X..........................................X.....] duration: 2.235721433s FAIL (2025-08-19 17:07:31 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 17:04:06.343826144 (263h38m0.774906265s) [upload=274088]: Starting upload Aug 19 17:04:06.343875278 (263h38m0.774955409s) [upload=274088]: Done starting upload Aug 19 17:04:06.343971703 (263h38m0.775051834s) [upload=274088]: Uploading segments... Aug 19 17:04:06.344028321 (263h38m0.775108453s) [upload=274088]: Got next segment. Inline: true Aug 19 17:04:06.344033120 (263h38m0.775113252s) [upload=274088]: Waiting for error group managing segments... Aug 19 17:04:06.344068187 (263h38m0.775148318s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:04:06.344071964 (263h38m0.775152095s) [upload=274088]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:04:06.344074499 (263h38m0.77515462s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:04:06.653287654 (263h38m1.084367785s) [upload=274088]: Done uploading segments... Aug 19 17:04:06.653300519 (263h38m1.08438065s) [upload=274088]: Upload finished. err: <nil> Duration:* 2.235752112s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X....X.........................] duration: 6.24541417s FAIL (2025-08-19 17:07:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.245432336s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [..X.X...X...X..............X......X.XX......................] duration: 5.800840948s FAIL (2025-08-19 17:07:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Aug 19 16:48:51.452491912 (263h22m45.883572043s) [upload=273834]: Starting upload Aug 19 16:48:51.452531028 (263h22m45.883611159s) [upload=273834]: Done starting upload Aug 19 16:48:51.452781006 (263h22m45.883861137s) [upload=273834]: Uploading segments... Aug 19 16:48:51.452853734 (263h22m45.883933866s) [upload=273834]: Got next segment. Inline: false Aug 19 16:48:51.452892670 (263h22m45.883972871s) [upload=273834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:48:51.452899813 (263h22m45.883980004s) [upload=273834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:48:51.962807465 (263h22m46.393887596s) [seg_pos={0 0}, upload=273834]: Begin upload segment... Aug 19 16:48:51.962852621 (263h22m46.393932743s) [seg_pos={0 0}, upload=273834]: Done begin upload segment. Aug 19 16:48:51.962875075 (263h22m46.393955206s) [noise=true , address=104.36.82.23:4470 , piece_id=5PDSSXU4HWGHBN62JJQBKHDQ6YYJVBWZZYSCJGKOPTZAZZUCAB6Q, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.962917746 (263h22m46.393997877s) [noise=true , address=66.23.193.175:4556 , piece_id=FBS7KZJF3HULYZZJDNLNOKMR5HSCR224IY7ZU4FUKZC5YMIIOAJA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963271022 (263h22m46.394351153s) [noise=true , address=45.91.116.9:4286 , piece_id=WYJJULBB3QVG2F46C52OCJWAMHVORGI7MDQS3CSYLF7REGAZQ3OQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963372556 (263h22m46.394452687s) [noise=true , address=156.146.43.225:4452 , piece_id=3T6HWTUPTX6A7BZULB4RSKGUO6GNDIHQGUP5W3EESJBWZGN7LLEQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963580684 (263h22m46.394660815s) [noise=true , address=104.36.82.35:4606 , piece_id=R37F22AIPFPV45K2PRWAJNONXQEDBJQPZGQKQ3LSHAQIBPKRXLHA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963653934 (263h22m46.394734066s) [noise=true , address=104.36.82.21:4478 , piece_id=XJWKXZ45KPRM3QUYXVIYU75M24GOCWGG5E3K4V62225XF3TGSS4Q, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963841874 (263h22m46.394922005s) [noise=true , address=79.127.201.215:4343 , piece_id=JF5HESY3ZARRZWZJHGTIG2BEZWWMXPHNJF5ALABQBAOVGJCHKI5A, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963910455 (263h22m46.394990586s) [noise=true , address=79.127.219.51:4211 , piece_id=OUB6QKVWCRJATMQFLAQ7UK3LJ4DILSLRAGFDP5YXROEXHZCPWGZA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964097293 (263h22m46.395177424s) [noise=true , address=104.36.82.60:4341 , piece_id=JCAHPCHE5IPHX35ZCDQ27I3DZ3R347HFRQWNBCNBXU6P3O7PTONA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964163640 (263h22m46.395243771s) [noise=true , address=77.90.31.18:4828 , piece_id=NFXT5LHIVX4G6DXMB4UWI37XCXLNHBRLVLXSJWTYAYN2JKCHRZQA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964400383 (263h22m46.395480514s) [noise=true , address=104.36.82.51:4052 , piece_id=7HLVSHJCDKI3TNKFETBUAY25HVIOPKAMSYQ6Q5BJ2NCJTQOTETTA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964624241 (263h22m46.395704372s) [noise=true , address=77.90.31.10:4529 , piece_id=HK5DLHGNULVF2RX7NNKFYN5JIFJB3LDEYXLXO3UN4D5OO4DJOW4A, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964832199 (263h22m46.39591233s) [noise=true , address=45.91.116.11:4130 , piece_id=5IZILPAZ4UKTFE766TWKT672JHDPWIQJI3TMOTASODK7NYG5BKPQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965048673 (263h22m46.396128804s) [noise=true , address=104.36.82.11:4127 , piece_id=6ACOOH24TTEMA4OQP6OFNHIABWDW3NDUVD44FSRIFLPTSVPHXOOA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965259626 (263h22m46.396339747s) [noise=true , address=77.90.31.15:4713 , piece_id=26S5ETN4OVNEL7IYYAIUBLP6UDS2QSSXCHD4H6YNHEQNY6FPOBOA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965452415 (263h22m46.396532546s) [noise=true , address=45.91.116.14:4209 , piece_id=5KAIVWUEWAG5NW6QGOAJLIMUADHZ6R4YTNAKTTXAGBRUEYXDAE7A, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965641637 (263h22m46.396721768s) [noise=true , address=45.91.116.7:4020 , piece_id=5NNTOPQ5BC7T2TLQ7NWSO3E6R4NDPB3JPXPMPRGG6I7YJU6QS4CQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965869794 (263h22m46.396949925s) [noise=true , address=77.90.31.17:4793 , piece_id=K3GCAOBIKBJV6Q3KOFXL5SEUFYE46MKJU5IAYNDA25IDCICTOFCQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:53.162668249 (263h22m47.59374838s) [upload=273834]: Next returned nil segment Aug 19 16:48:53.162683478 (263h22m47.593763599s) [upload=273834]: Waiting for error group managing segments... Aug 19 16:48:53.488132069 (263h22m47.91921223s) [noise=true , address=66.23.193.175:4556 , piece_id=FBS7KZJF3HULYZZJDNLNOKMR5HSCR224IY7ZU4FUKZC5YMIIOAJA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.887114541 (263h22m48.318194672s) [noise=true , address=79.127.219.51:4211 , piece_id=OUB6QKVWCRJATMQFLAQ7UK3LJ4DILSLRAGFDP5YXROEXHZCPWGZA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.925197211 (263h22m48.356277342s) [noise=true , address=156.146.43.225:4452 , piece_id=3T6HWTUPTX6A7BZULB4RSKGUO6GNDIHQGUP5W3EESJBWZGN7LLEQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.961937701 (263h22m48.393017832s) [noise=true , address=104.36.82.11:4127 , piece_id=6ACOOH24TTEMA4OQP6OFNHIABWDW3NDUVD44FSRIFLPTSVPHXOOA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.992310049 (263h22m48.42339018s) [noise=true , address=79.127.201.215:4343 , piece_id=JF5HESY3ZARRZWZJHGTIG2BEZWWMXPHNJF5ALABQBAOVGJCHKI5A, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.045243820 (263h22m48.476323961s) [noise=true , address=104.36.82.35:4606 , piece_id=R37F22AIPFPV45K2PRWAJNONXQEDBJQPZGQKQ3LSHAQIBPKRXLHA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.059875193 (263h22m48.490955324s) [noise=true , address=104.36.82.21:4478 , piece_id=XJWKXZ45KPRM3QUYXVIYU75M24GOCWGG5E3K4V62225XF3TGSS4Q, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.095629989 (263h22m48.52671012s) [noise=true , address=104.36.82.23:4470 , piece_id=5PDSSXU4HWGHBN62JJQBKHDQ6YYJVBWZZYSCJGKOPTZAZZUCAB6Q, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.103250568 (263h22m48.534330699s) [noise=true , address=104.36.82.51:4052 , piece_id=7HLVSHJCDKI3TNKFETBUAY25HVIOPKAMSYQ6Q5BJ2NCJTQOTETTA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.159963800 (263h22m48.591043921s) [noise=true , address=104.36.82.60:4341 , piece_id=JCAHPCHE5IPHX35ZCDQ27I3DZ3R347HFRQWNBCNBXU6P3O7PTONA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:49:37.214809078 (263h23m31.64588921s) [noise=true , address=77.90.31.17:4793 , piece_id=K3GCAOBIKBJV6Q3KOFXL5SEUFYE46MKJU5IAYNDA25IDCICTOFCQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:49:50.811468753 (263h23m45.242548894s) [noise=true , address=77.90.31.10:4529 , piece_id=HK5DLHGNULVF2RX7NNKFYN5JIFJB3LDEYXLXO3UN4D5OO4DJOW4A, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: read tcp 188.166.225.170:57426->77.90.31.10:4529: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811576489 (263h23m45.24265662s) [noise=true , address=45.91.116.14:4209 , piece_id=5KAIVWUEWAG5NW6QGOAJLIMUADHZ6R4YTNAKTTXAGBRUEYXDAE7A, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:12LWbAxsrDcyV31SxJNiVNYs7ERKAf6RTuKpE596UrYJAnXMndR, address:45.91.116.14:4209): protocol: rpc: dial tcp 45.91.116.14:4209: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811688974 (263h23m45.242769105s) [noise=true , address=45.91.116.9:4286 , piece_id=WYJJULBB3QVG2F46C52OCJWAMHVORGI7MDQS3CSYLF7REGAZQ3OQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:19DfbHig9tzF4VRHtSWkExwUS2bdZ8GFgKnAD9wvMBG7qaesNa, address:45.91.116.9:4286): protocol: read tcp 188.166.225.170:40032->45.91.116.9:4286: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811727057 (263h23m45.242807188s) [noise=true , address=45.91.116.11:4130 , piece_id=5IZILPAZ4UKTFE766TWKT672JHDPWIQJI3TMOTASODK7NYG5BKPQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:126ThYkfUAYf2JR8f5rkwyumcvmRArxKbpLUjhJukoc3FADHj2T, address:45.91.116.11:4130): protocol: read tcp 188.166.225.170:46860->45.91.116.11:4130: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811808934 (263h23m45.242889085s) [noise=true , address=45.91.116.7:4020 , piece_id=5NNTOPQ5BC7T2TLQ7NWSO3E6R4NDPB3JPXPMPRGG6I7YJU6QS4CQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:1MT5Nk9DRHLLfHi5sJnzWEjuL3aV2HnbS9Xebdw8vtBMjQmFnV, address:45.91.116.7:4020): protocol: read tcp 188.166.225.170:48966->45.91.116.7:4020: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811874559 (263h23m45.24295469s) [noise=true , address=77.90.31.18:4828 , piece_id=NFXT5LHIVX4G6DXMB4UWI37XCXLNHBRLVLXSJWTYAYN2JKCHRZQA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:1jwTDXPShXfNdfMx3UXi22gQphBpFre1MhRsHyjq6xRz4wwTr9, address:77.90.31.18:4828): protocol: read tcp 188.166.225.170:47612->77.90.31.18:4828: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811914556 (263h23m45.242994687s) [noise=true , address=77.90.31.15:4713 , piece_id=26S5ETN4OVNEL7IYYAIUBLP6UDS2QSSXCHD4H6YNHEQNY6FPOBOA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: read tcp 188.166.225.170:41506->77.90.31.15:4713: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811942720 (263h23m45.243022841s) [seg_pos={0 0}, upload=273834]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Aug 19 16:49:51.082762534 (263h23m45.513842665s) [upload=273834]: Done uploading segments... Aug 19 16:49:51.082792671 (263h23m45.513872803s) [upload=273834]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.800865254s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............................X...X..........................] duration: 6.484273612s FAIL (2025-08-19 17:07:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.484304643s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................X....XX.........................] duration: 6.118735324s FAIL (2025-08-19 17:06:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 19 16:49:12.801154313 (263h23m7.232234445s) [upload=273839]: Starting upload Aug 19 16:49:12.801215942 (263h23m7.232296073s) [upload=273839]: Done starting upload Aug 19 16:49:12.801330351 (263h23m7.232410482s) [upload=273839]: Uploading segments... Aug 19 16:49:12.801425372 (263h23m7.232505503s) [upload=273839]: Got next segment. Inline: false Aug 19 16:49:12.801455880 (263h23m7.232536012s) [upload=273839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:49:12.801461982 (263h23m7.232542113s) [upload=273839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:49:13.036708228 (263h23m7.467788359s) [seg_pos={0 0}, upload=273839]: Begin upload segment... Aug 19 16:49:13.036749197 (263h23m7.467829328s) [seg_pos={0 0}, upload=273839]: Done begin upload segment. Aug 19 16:49:13.036771971 (263h23m7.467852092s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.036780256 (263h23m7.467860437s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037135917 (263h23m7.468216048s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037326592 (263h23m7.468406723s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037403259 (263h23m7.46848339s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037667814 (263h23m7.468747945s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037761764 (263h23m7.468841896s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037943331 (263h23m7.469023462s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038070565 (263h23m7.469150696s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038168622 (263h23m7.469248753s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038300104 (263h23m7.469380235s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038396789 (263h23m7.46947692s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038622241 (263h23m7.469702362s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038679350 (263h23m7.469759481s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038854735 (263h23m7.469934856s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.039048045 (263h23m7.470128166s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:14.749064365 (263h23m9.180144496s) [upload=273839]: Next returned nil segment Aug 19 16:49:14.749117217 (263h23m9.180197348s) [upload=273839]: Waiting for error group managing segments... Aug 19 16:49:15.090317863 (263h23m9.521397994s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.132109007 (263h23m9.563189138s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.222090156 (263h23m9.653170287s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.328041780 (263h23m9.759121911s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.353998781 (263h23m9.785078912s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:36.709543777 (263h23m31.140623898s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:37.041062020 (263h23m31.472142141s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.183707791 (263h23m32.614787922s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.694075084 (263h23m33.125155215s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.716788163 (263h23m34.147868294s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.851890717 (263h23m34.282970848s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:40.293306092 (263h23m34.724386243s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:42.156929229 (263h23m36.58800936s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157075889 (263h23m36.58815601s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157126075 (263h23m36.588206206s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection Aug 19 16:49:42.157170951 (263h23m36.588251082s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: read tcp 188.166.225.170:34242->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:42.157215747 (263h23m36.588295878s) [seg_pos={0 0}, upload=273839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 19 16:49:42.385478364 (263h23m36.816558495s) [upload=273839]: Done uploading segments... Aug 19 16:49:42.385503091 (263h23m36.816583212s) [upload=273839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.118765351s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.722481797s FAIL (2025-08-19 17:06:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 263h18m33.75518017s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324344006 (263h18m33.755424127s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324633220 (263h18m33.755713351s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324851286 (263h18m33.755931417s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325068942 (263h18m33.756149063s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325283613 (263h18m33.756363744s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325499857 (263h18m33.756579998s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325759794 (263h18m33.756839915s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326059428 (263h18m33.757139549s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326277565 (263h18m33.757357696s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326470835 (263h18m33.757550966s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326673753 (263h18m33.757753884s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326867664 (263h18m33.757947795s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:40.851885389 (263h18m35.28296552s) [upload=273764]: Next returned nil segment Aug 19 16:44:40.851903334 (263h18m35.282983465s) [upload=273764]: Waiting for error group managing segments... Aug 19 16:44:41.244668372 (263h18m35.675748503s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.277280365 (263h18m35.708360496s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.328561665 (263h18m35.759641786s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.331747139 (263h18m35.76282727s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.357720320 (263h18m35.788800481s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.378190805 (263h18m35.809270936s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.486480333 (263h18m35.917560464s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:45:38.230570787 (263h19m32.661650918s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: expected piece hash; context deadline exceeded; EOF Aug 19 16:45:38.230979869 (263h19m32.66206s) [noise=true , address=77.90.31.18:4840 , piece_id=JWOETSFSZ3HELLZNLEASYNGW5OR64TMQVOFYYJAJVTHWCBHVTSLQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: read tcp 188.166.225.170:39946->77.90.31.18:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231017812 (263h19m32.662097943s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 188.166.225.170:44164->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231063489 (263h19m32.66214362s) [noise=true , address=45.91.116.14:4203 , piece_id=WRLBES3FI5EA5AQLZXJYGDROP3KBG3E2F22R4YD6A5KKI44PLV2Q, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 188.166.225.170:51738->45.91.116.14:4203: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231090952 (263h19m32.662171083s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: read tcp 188.166.225.170:57158->77.90.31.15:4707: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231110719 (263h19m32.66219085s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:50784->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231140165 (263h19m32.662220296s) [noise=true , address=77.90.31.16:4749 , piece_id=XCPXVBYLQH345FPIW4KDNLBFUCDDEW2H4B25WIXGOO5FVMAIVKPQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: read tcp 188.166.225.170:51668->77.90.31.16:4749: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231144814 (263h19m32.662224945s) [noise=true , address=77.90.31.19:4865 , piece_id=B3HUFIJZKVZJH4OBBO2DD7S3I7HX4LO2IQ74XJPO4DTW4QER35NA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1bguBghQonV5sTHaWteyUEnpp5KNzBKbvBpbvrUNFDCveHN77X, address:77.90.31.19:4865): protocol: read tcp 188.166.225.170:39398->77.90.31.19:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231177167 (263h19m32.662257298s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: read tcp 188.166.225.170:58526->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231184571 (263h19m32.662264702s) [noise=true , address=45.91.116.12:4236 , piece_id=LCRQS2GOSHOSLZBGQTGB3Q3PA4VN7KXNW66TDZ5O424TPJWTQE3A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH, address:45.91.116.12:4236): protocol: read tcp 188.166.225.170:57166->45.91.116.12:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231189770 (263h19m32.662269901s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 188.166.225.170:54242->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231215850 (263h19m32.662295981s) [seg_pos={0 0}, upload=273764]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 19 16:45:38.459100173 (263h19m32.890180304s) [upload=273764]: Done uploading segments... Aug 19 16:45:38.459128316 (263h19m32.890208447s) [upload=273764]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.722505183s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [............X..........................................X....] duration: 2.025964125s FAIL (2025-08-19 17:06:49 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 17:04:06.343826144 (263h38m0.774906265s) [upload=274088]: Starting upload Aug 19 17:04:06.343875278 (263h38m0.774955409s) [upload=274088]: Done starting upload Aug 19 17:04:06.343971703 (263h38m0.775051834s) [upload=274088]: Uploading segments... Aug 19 17:04:06.344028321 (263h38m0.775108453s) [upload=274088]: Got next segment. Inline: true Aug 19 17:04:06.344033120 (263h38m0.775113252s) [upload=274088]: Waiting for error group managing segments... Aug 19 17:04:06.344068187 (263h38m0.775148318s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:04:06.344071964 (263h38m0.775152095s) [upload=274088]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:04:06.344074499 (263h38m0.77515462s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:04:06.653287654 (263h38m1.084367785s) [upload=274088]: Done uploading segments... Aug 19 17:04:06.653300519 (263h38m1.08438065s) [upload=274088]: Upload finished. err: <nil> Duration:* 2.025986778s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X....X........................] duration: 6.124513459s FAIL (2025-08-19 17:06:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.124614853s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.712770169s FAIL (2025-08-19 17:06:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 263h18m33.75518017s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324344006 (263h18m33.755424127s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324633220 (263h18m33.755713351s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324851286 (263h18m33.755931417s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325068942 (263h18m33.756149063s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325283613 (263h18m33.756363744s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325499857 (263h18m33.756579998s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325759794 (263h18m33.756839915s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326059428 (263h18m33.757139549s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326277565 (263h18m33.757357696s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326470835 (263h18m33.757550966s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326673753 (263h18m33.757753884s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326867664 (263h18m33.757947795s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:40.851885389 (263h18m35.28296552s) [upload=273764]: Next returned nil segment Aug 19 16:44:40.851903334 (263h18m35.282983465s) [upload=273764]: Waiting for error group managing segments... Aug 19 16:44:41.244668372 (263h18m35.675748503s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.277280365 (263h18m35.708360496s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.328561665 (263h18m35.759641786s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.331747139 (263h18m35.76282727s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.357720320 (263h18m35.788800481s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.378190805 (263h18m35.809270936s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.486480333 (263h18m35.917560464s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:45:38.230570787 (263h19m32.661650918s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: expected piece hash; context deadline exceeded; EOF Aug 19 16:45:38.230979869 (263h19m32.66206s) [noise=true , address=77.90.31.18:4840 , piece_id=JWOETSFSZ3HELLZNLEASYNGW5OR64TMQVOFYYJAJVTHWCBHVTSLQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: read tcp 188.166.225.170:39946->77.90.31.18:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231017812 (263h19m32.662097943s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 188.166.225.170:44164->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231063489 (263h19m32.66214362s) [noise=true , address=45.91.116.14:4203 , piece_id=WRLBES3FI5EA5AQLZXJYGDROP3KBG3E2F22R4YD6A5KKI44PLV2Q, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 188.166.225.170:51738->45.91.116.14:4203: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231090952 (263h19m32.662171083s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: read tcp 188.166.225.170:57158->77.90.31.15:4707: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231110719 (263h19m32.66219085s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:50784->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231140165 (263h19m32.662220296s) [noise=true , address=77.90.31.16:4749 , piece_id=XCPXVBYLQH345FPIW4KDNLBFUCDDEW2H4B25WIXGOO5FVMAIVKPQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: read tcp 188.166.225.170:51668->77.90.31.16:4749: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231144814 (263h19m32.662224945s) [noise=true , address=77.90.31.19:4865 , piece_id=B3HUFIJZKVZJH4OBBO2DD7S3I7HX4LO2IQ74XJPO4DTW4QER35NA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1bguBghQonV5sTHaWteyUEnpp5KNzBKbvBpbvrUNFDCveHN77X, address:77.90.31.19:4865): protocol: read tcp 188.166.225.170:39398->77.90.31.19:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231177167 (263h19m32.662257298s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: read tcp 188.166.225.170:58526->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231184571 (263h19m32.662264702s) [noise=true , address=45.91.116.12:4236 , piece_id=LCRQS2GOSHOSLZBGQTGB3Q3PA4VN7KXNW66TDZ5O424TPJWTQE3A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH, address:45.91.116.12:4236): protocol: read tcp 188.166.225.170:57166->45.91.116.12:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231189770 (263h19m32.662269901s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 188.166.225.170:54242->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231215850 (263h19m32.662295981s) [seg_pos={0 0}, upload=273764]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 19 16:45:38.459100173 (263h19m32.890180304s) [upload=273764]: Done uploading segments... Aug 19 16:45:38.459128316 (263h19m32.890208447s) [upload=273764]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.712809384s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.............X..........................................X...] duration: 2.177314676s FAIL (2025-08-19 17:06:02 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 17:04:06.343826144 (263h38m0.774906265s) [upload=274088]: Starting upload Aug 19 17:04:06.343875278 (263h38m0.774955409s) [upload=274088]: Done starting upload Aug 19 17:04:06.343971703 (263h38m0.775051834s) [upload=274088]: Uploading segments... Aug 19 17:04:06.344028321 (263h38m0.775108453s) [upload=274088]: Got next segment. Inline: true Aug 19 17:04:06.344033120 (263h38m0.775113252s) [upload=274088]: Waiting for error group managing segments... Aug 19 17:04:06.344068187 (263h38m0.775148318s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:04:06.344071964 (263h38m0.775152095s) [upload=274088]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:04:06.344074499 (263h38m0.77515462s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:04:06.653287654 (263h38m1.084367785s) [upload=274088]: Done uploading segments... Aug 19 17:04:06.653300519 (263h38m1.08438065s) [upload=274088]: Upload finished. err: <nil> Duration:* 2.177366536s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [...X.X...X...X..............X......X.XX.....................] duration: 6.220613288s FAIL (2025-08-19 17:05:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Aug 19 16:48:51.452491912 (263h22m45.883572043s) [upload=273834]: Starting upload Aug 19 16:48:51.452531028 (263h22m45.883611159s) [upload=273834]: Done starting upload Aug 19 16:48:51.452781006 (263h22m45.883861137s) [upload=273834]: Uploading segments... Aug 19 16:48:51.452853734 (263h22m45.883933866s) [upload=273834]: Got next segment. Inline: false Aug 19 16:48:51.452892670 (263h22m45.883972871s) [upload=273834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:48:51.452899813 (263h22m45.883980004s) [upload=273834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:48:51.962807465 (263h22m46.393887596s) [seg_pos={0 0}, upload=273834]: Begin upload segment... Aug 19 16:48:51.962852621 (263h22m46.393932743s) [seg_pos={0 0}, upload=273834]: Done begin upload segment. Aug 19 16:48:51.962875075 (263h22m46.393955206s) [noise=true , address=104.36.82.23:4470 , piece_id=5PDSSXU4HWGHBN62JJQBKHDQ6YYJVBWZZYSCJGKOPTZAZZUCAB6Q, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.962917746 (263h22m46.393997877s) [noise=true , address=66.23.193.175:4556 , piece_id=FBS7KZJF3HULYZZJDNLNOKMR5HSCR224IY7ZU4FUKZC5YMIIOAJA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963271022 (263h22m46.394351153s) [noise=true , address=45.91.116.9:4286 , piece_id=WYJJULBB3QVG2F46C52OCJWAMHVORGI7MDQS3CSYLF7REGAZQ3OQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963372556 (263h22m46.394452687s) [noise=true , address=156.146.43.225:4452 , piece_id=3T6HWTUPTX6A7BZULB4RSKGUO6GNDIHQGUP5W3EESJBWZGN7LLEQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963580684 (263h22m46.394660815s) [noise=true , address=104.36.82.35:4606 , piece_id=R37F22AIPFPV45K2PRWAJNONXQEDBJQPZGQKQ3LSHAQIBPKRXLHA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963653934 (263h22m46.394734066s) [noise=true , address=104.36.82.21:4478 , piece_id=XJWKXZ45KPRM3QUYXVIYU75M24GOCWGG5E3K4V62225XF3TGSS4Q, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963841874 (263h22m46.394922005s) [noise=true , address=79.127.201.215:4343 , piece_id=JF5HESY3ZARRZWZJHGTIG2BEZWWMXPHNJF5ALABQBAOVGJCHKI5A, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963910455 (263h22m46.394990586s) [noise=true , address=79.127.219.51:4211 , piece_id=OUB6QKVWCRJATMQFLAQ7UK3LJ4DILSLRAGFDP5YXROEXHZCPWGZA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964097293 (263h22m46.395177424s) [noise=true , address=104.36.82.60:4341 , piece_id=JCAHPCHE5IPHX35ZCDQ27I3DZ3R347HFRQWNBCNBXU6P3O7PTONA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964163640 (263h22m46.395243771s) [noise=true , address=77.90.31.18:4828 , piece_id=NFXT5LHIVX4G6DXMB4UWI37XCXLNHBRLVLXSJWTYAYN2JKCHRZQA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964400383 (263h22m46.395480514s) [noise=true , address=104.36.82.51:4052 , piece_id=7HLVSHJCDKI3TNKFETBUAY25HVIOPKAMSYQ6Q5BJ2NCJTQOTETTA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964624241 (263h22m46.395704372s) [noise=true , address=77.90.31.10:4529 , piece_id=HK5DLHGNULVF2RX7NNKFYN5JIFJB3LDEYXLXO3UN4D5OO4DJOW4A, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964832199 (263h22m46.39591233s) [noise=true , address=45.91.116.11:4130 , piece_id=5IZILPAZ4UKTFE766TWKT672JHDPWIQJI3TMOTASODK7NYG5BKPQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965048673 (263h22m46.396128804s) [noise=true , address=104.36.82.11:4127 , piece_id=6ACOOH24TTEMA4OQP6OFNHIABWDW3NDUVD44FSRIFLPTSVPHXOOA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965259626 (263h22m46.396339747s) [noise=true , address=77.90.31.15:4713 , piece_id=26S5ETN4OVNEL7IYYAIUBLP6UDS2QSSXCHD4H6YNHEQNY6FPOBOA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965452415 (263h22m46.396532546s) [noise=true , address=45.91.116.14:4209 , piece_id=5KAIVWUEWAG5NW6QGOAJLIMUADHZ6R4YTNAKTTXAGBRUEYXDAE7A, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965641637 (263h22m46.396721768s) [noise=true , address=45.91.116.7:4020 , piece_id=5NNTOPQ5BC7T2TLQ7NWSO3E6R4NDPB3JPXPMPRGG6I7YJU6QS4CQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965869794 (263h22m46.396949925s) [noise=true , address=77.90.31.17:4793 , piece_id=K3GCAOBIKBJV6Q3KOFXL5SEUFYE46MKJU5IAYNDA25IDCICTOFCQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:53.162668249 (263h22m47.59374838s) [upload=273834]: Next returned nil segment Aug 19 16:48:53.162683478 (263h22m47.593763599s) [upload=273834]: Waiting for error group managing segments... Aug 19 16:48:53.488132069 (263h22m47.91921223s) [noise=true , address=66.23.193.175:4556 , piece_id=FBS7KZJF3HULYZZJDNLNOKMR5HSCR224IY7ZU4FUKZC5YMIIOAJA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.887114541 (263h22m48.318194672s) [noise=true , address=79.127.219.51:4211 , piece_id=OUB6QKVWCRJATMQFLAQ7UK3LJ4DILSLRAGFDP5YXROEXHZCPWGZA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.925197211 (263h22m48.356277342s) [noise=true , address=156.146.43.225:4452 , piece_id=3T6HWTUPTX6A7BZULB4RSKGUO6GNDIHQGUP5W3EESJBWZGN7LLEQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.961937701 (263h22m48.393017832s) [noise=true , address=104.36.82.11:4127 , piece_id=6ACOOH24TTEMA4OQP6OFNHIABWDW3NDUVD44FSRIFLPTSVPHXOOA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.992310049 (263h22m48.42339018s) [noise=true , address=79.127.201.215:4343 , piece_id=JF5HESY3ZARRZWZJHGTIG2BEZWWMXPHNJF5ALABQBAOVGJCHKI5A, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.045243820 (263h22m48.476323961s) [noise=true , address=104.36.82.35:4606 , piece_id=R37F22AIPFPV45K2PRWAJNONXQEDBJQPZGQKQ3LSHAQIBPKRXLHA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.059875193 (263h22m48.490955324s) [noise=true , address=104.36.82.21:4478 , piece_id=XJWKXZ45KPRM3QUYXVIYU75M24GOCWGG5E3K4V62225XF3TGSS4Q, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.095629989 (263h22m48.52671012s) [noise=true , address=104.36.82.23:4470 , piece_id=5PDSSXU4HWGHBN62JJQBKHDQ6YYJVBWZZYSCJGKOPTZAZZUCAB6Q, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.103250568 (263h22m48.534330699s) [noise=true , address=104.36.82.51:4052 , piece_id=7HLVSHJCDKI3TNKFETBUAY25HVIOPKAMSYQ6Q5BJ2NCJTQOTETTA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.159963800 (263h22m48.591043921s) [noise=true , address=104.36.82.60:4341 , piece_id=JCAHPCHE5IPHX35ZCDQ27I3DZ3R347HFRQWNBCNBXU6P3O7PTONA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:49:37.214809078 (263h23m31.64588921s) [noise=true , address=77.90.31.17:4793 , piece_id=K3GCAOBIKBJV6Q3KOFXL5SEUFYE46MKJU5IAYNDA25IDCICTOFCQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:49:50.811468753 (263h23m45.242548894s) [noise=true , address=77.90.31.10:4529 , piece_id=HK5DLHGNULVF2RX7NNKFYN5JIFJB3LDEYXLXO3UN4D5OO4DJOW4A, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: read tcp 188.166.225.170:57426->77.90.31.10:4529: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811576489 (263h23m45.24265662s) [noise=true , address=45.91.116.14:4209 , piece_id=5KAIVWUEWAG5NW6QGOAJLIMUADHZ6R4YTNAKTTXAGBRUEYXDAE7A, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:12LWbAxsrDcyV31SxJNiVNYs7ERKAf6RTuKpE596UrYJAnXMndR, address:45.91.116.14:4209): protocol: rpc: dial tcp 45.91.116.14:4209: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811688974 (263h23m45.242769105s) [noise=true , address=45.91.116.9:4286 , piece_id=WYJJULBB3QVG2F46C52OCJWAMHVORGI7MDQS3CSYLF7REGAZQ3OQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:19DfbHig9tzF4VRHtSWkExwUS2bdZ8GFgKnAD9wvMBG7qaesNa, address:45.91.116.9:4286): protocol: read tcp 188.166.225.170:40032->45.91.116.9:4286: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811727057 (263h23m45.242807188s) [noise=true , address=45.91.116.11:4130 , piece_id=5IZILPAZ4UKTFE766TWKT672JHDPWIQJI3TMOTASODK7NYG5BKPQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:126ThYkfUAYf2JR8f5rkwyumcvmRArxKbpLUjhJukoc3FADHj2T, address:45.91.116.11:4130): protocol: read tcp 188.166.225.170:46860->45.91.116.11:4130: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811808934 (263h23m45.242889085s) [noise=true , address=45.91.116.7:4020 , piece_id=5NNTOPQ5BC7T2TLQ7NWSO3E6R4NDPB3JPXPMPRGG6I7YJU6QS4CQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:1MT5Nk9DRHLLfHi5sJnzWEjuL3aV2HnbS9Xebdw8vtBMjQmFnV, address:45.91.116.7:4020): protocol: read tcp 188.166.225.170:48966->45.91.116.7:4020: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811874559 (263h23m45.24295469s) [noise=true , address=77.90.31.18:4828 , piece_id=NFXT5LHIVX4G6DXMB4UWI37XCXLNHBRLVLXSJWTYAYN2JKCHRZQA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:1jwTDXPShXfNdfMx3UXi22gQphBpFre1MhRsHyjq6xRz4wwTr9, address:77.90.31.18:4828): protocol: read tcp 188.166.225.170:47612->77.90.31.18:4828: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811914556 (263h23m45.242994687s) [noise=true , address=77.90.31.15:4713 , piece_id=26S5ETN4OVNEL7IYYAIUBLP6UDS2QSSXCHD4H6YNHEQNY6FPOBOA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: read tcp 188.166.225.170:41506->77.90.31.15:4713: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811942720 (263h23m45.243022841s) [seg_pos={0 0}, upload=273834]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Aug 19 16:49:51.082762534 (263h23m45.513842665s) [upload=273834]: Done uploading segments... Aug 19 16:49:51.082792671 (263h23m45.513872803s) [upload=273834]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 6.220632856s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............................X...X.........................] duration: 6.541447628s FAIL (2025-08-19 17:05:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.541472526s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..............X..........................................X..] duration: 2.29360062s FAIL (2025-08-19 17:05:41 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 17:04:06.343826144 (263h38m0.774906265s) [upload=274088]: Starting upload Aug 19 17:04:06.343875278 (263h38m0.774955409s) [upload=274088]: Done starting upload Aug 19 17:04:06.343971703 (263h38m0.775051834s) [upload=274088]: Uploading segments... Aug 19 17:04:06.344028321 (263h38m0.775108453s) [upload=274088]: Got next segment. Inline: true Aug 19 17:04:06.344033120 (263h38m0.775113252s) [upload=274088]: Waiting for error group managing segments... Aug 19 17:04:06.344068187 (263h38m0.775148318s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:04:06.344071964 (263h38m0.775152095s) [upload=274088]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:04:06.344074499 (263h38m0.77515462s) [upload=274088]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:04:06.653287654 (263h38m1.084367785s) [upload=274088]: Done uploading segments... Aug 19 17:04:06.653300519 (263h38m1.08438065s) [upload=274088]: Upload finished. err: <nil> Duration:* 2.293628202s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................X....XX........................] duration: 5.778374163s FAIL (2025-08-19 17:05:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 19 16:49:12.801154313 (263h23m7.232234445s) [upload=273839]: Starting upload Aug 19 16:49:12.801215942 (263h23m7.232296073s) [upload=273839]: Done starting upload Aug 19 16:49:12.801330351 (263h23m7.232410482s) [upload=273839]: Uploading segments... Aug 19 16:49:12.801425372 (263h23m7.232505503s) [upload=273839]: Got next segment. Inline: false Aug 19 16:49:12.801455880 (263h23m7.232536012s) [upload=273839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:49:12.801461982 (263h23m7.232542113s) [upload=273839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:49:13.036708228 (263h23m7.467788359s) [seg_pos={0 0}, upload=273839]: Begin upload segment... Aug 19 16:49:13.036749197 (263h23m7.467829328s) [seg_pos={0 0}, upload=273839]: Done begin upload segment. Aug 19 16:49:13.036771971 (263h23m7.467852092s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.036780256 (263h23m7.467860437s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037135917 (263h23m7.468216048s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037326592 (263h23m7.468406723s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037403259 (263h23m7.46848339s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037667814 (263h23m7.468747945s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037761764 (263h23m7.468841896s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037943331 (263h23m7.469023462s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038070565 (263h23m7.469150696s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038168622 (263h23m7.469248753s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038300104 (263h23m7.469380235s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038396789 (263h23m7.46947692s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038622241 (263h23m7.469702362s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038679350 (263h23m7.469759481s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038854735 (263h23m7.469934856s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.039048045 (263h23m7.470128166s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:14.749064365 (263h23m9.180144496s) [upload=273839]: Next returned nil segment Aug 19 16:49:14.749117217 (263h23m9.180197348s) [upload=273839]: Waiting for error group managing segments... Aug 19 16:49:15.090317863 (263h23m9.521397994s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.132109007 (263h23m9.563189138s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.222090156 (263h23m9.653170287s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.328041780 (263h23m9.759121911s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.353998781 (263h23m9.785078912s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:36.709543777 (263h23m31.140623898s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:37.041062020 (263h23m31.472142141s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.183707791 (263h23m32.614787922s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.694075084 (263h23m33.125155215s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.716788163 (263h23m34.147868294s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.851890717 (263h23m34.282970848s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:40.293306092 (263h23m34.724386243s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:42.156929229 (263h23m36.58800936s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157075889 (263h23m36.58815601s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157126075 (263h23m36.588206206s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection Aug 19 16:49:42.157170951 (263h23m36.588251082s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: read tcp 188.166.225.170:34242->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:42.157215747 (263h23m36.588295878s) [seg_pos={0 0}, upload=273839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 19 16:49:42.385478364 (263h23m36.816558495s) [upload=273839]: Done uploading segments... Aug 19 16:49:42.385503091 (263h23m36.816583212s) [upload=273839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.7784043s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)