Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................X....XX.............] duration: 5.988445162s FAIL (2025-08-19 16:57:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 19 16:49:12.801154313 (263h23m7.232234445s) [upload=273839]: Starting upload Aug 19 16:49:12.801215942 (263h23m7.232296073s) [upload=273839]: Done starting upload Aug 19 16:49:12.801330351 (263h23m7.232410482s) [upload=273839]: Uploading segments... Aug 19 16:49:12.801425372 (263h23m7.232505503s) [upload=273839]: Got next segment. Inline: false Aug 19 16:49:12.801455880 (263h23m7.232536012s) [upload=273839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:49:12.801461982 (263h23m7.232542113s) [upload=273839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:49:13.036708228 (263h23m7.467788359s) [seg_pos={0 0}, upload=273839]: Begin upload segment... Aug 19 16:49:13.036749197 (263h23m7.467829328s) [seg_pos={0 0}, upload=273839]: Done begin upload segment. Aug 19 16:49:13.036771971 (263h23m7.467852092s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.036780256 (263h23m7.467860437s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037135917 (263h23m7.468216048s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037326592 (263h23m7.468406723s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037403259 (263h23m7.46848339s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037667814 (263h23m7.468747945s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037761764 (263h23m7.468841896s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037943331 (263h23m7.469023462s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038070565 (263h23m7.469150696s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038168622 (263h23m7.469248753s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038300104 (263h23m7.469380235s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038396789 (263h23m7.46947692s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038622241 (263h23m7.469702362s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038679350 (263h23m7.469759481s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038854735 (263h23m7.469934856s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.039048045 (263h23m7.470128166s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:14.749064365 (263h23m9.180144496s) [upload=273839]: Next returned nil segment Aug 19 16:49:14.749117217 (263h23m9.180197348s) [upload=273839]: Waiting for error group managing segments... Aug 19 16:49:15.090317863 (263h23m9.521397994s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.132109007 (263h23m9.563189138s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.222090156 (263h23m9.653170287s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.328041780 (263h23m9.759121911s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.353998781 (263h23m9.785078912s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:36.709543777 (263h23m31.140623898s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:37.041062020 (263h23m31.472142141s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.183707791 (263h23m32.614787922s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.694075084 (263h23m33.125155215s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.716788163 (263h23m34.147868294s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.851890717 (263h23m34.282970848s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:40.293306092 (263h23m34.724386243s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:42.156929229 (263h23m36.58800936s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157075889 (263h23m36.58815601s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157126075 (263h23m36.588206206s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection Aug 19 16:49:42.157170951 (263h23m36.588251082s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: read tcp 188.166.225.170:34242->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:42.157215747 (263h23m36.588295878s) [seg_pos={0 0}, upload=273839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 19 16:49:42.385478364 (263h23m36.816558495s) [upload=273839]: Done uploading segments... Aug 19 16:49:42.385503091 (263h23m36.816583212s) [upload=273839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.988468575s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.998933519s FAIL (2025-08-19 16:57:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 263h18m33.75518017s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324344006 (263h18m33.755424127s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324633220 (263h18m33.755713351s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324851286 (263h18m33.755931417s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325068942 (263h18m33.756149063s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325283613 (263h18m33.756363744s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325499857 (263h18m33.756579998s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325759794 (263h18m33.756839915s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326059428 (263h18m33.757139549s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326277565 (263h18m33.757357696s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326470835 (263h18m33.757550966s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326673753 (263h18m33.757753884s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326867664 (263h18m33.757947795s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:40.851885389 (263h18m35.28296552s) [upload=273764]: Next returned nil segment Aug 19 16:44:40.851903334 (263h18m35.282983465s) [upload=273764]: Waiting for error group managing segments... Aug 19 16:44:41.244668372 (263h18m35.675748503s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.277280365 (263h18m35.708360496s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.328561665 (263h18m35.759641786s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.331747139 (263h18m35.76282727s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.357720320 (263h18m35.788800481s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.378190805 (263h18m35.809270936s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.486480333 (263h18m35.917560464s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:45:38.230570787 (263h19m32.661650918s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: expected piece hash; context deadline exceeded; EOF Aug 19 16:45:38.230979869 (263h19m32.66206s) [noise=true , address=77.90.31.18:4840 , piece_id=JWOETSFSZ3HELLZNLEASYNGW5OR64TMQVOFYYJAJVTHWCBHVTSLQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: read tcp 188.166.225.170:39946->77.90.31.18:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231017812 (263h19m32.662097943s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 188.166.225.170:44164->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231063489 (263h19m32.66214362s) [noise=true , address=45.91.116.14:4203 , piece_id=WRLBES3FI5EA5AQLZXJYGDROP3KBG3E2F22R4YD6A5KKI44PLV2Q, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 188.166.225.170:51738->45.91.116.14:4203: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231090952 (263h19m32.662171083s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: read tcp 188.166.225.170:57158->77.90.31.15:4707: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231110719 (263h19m32.66219085s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:50784->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231140165 (263h19m32.662220296s) [noise=true , address=77.90.31.16:4749 , piece_id=XCPXVBYLQH345FPIW4KDNLBFUCDDEW2H4B25WIXGOO5FVMAIVKPQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: read tcp 188.166.225.170:51668->77.90.31.16:4749: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231144814 (263h19m32.662224945s) [noise=true , address=77.90.31.19:4865 , piece_id=B3HUFIJZKVZJH4OBBO2DD7S3I7HX4LO2IQ74XJPO4DTW4QER35NA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1bguBghQonV5sTHaWteyUEnpp5KNzBKbvBpbvrUNFDCveHN77X, address:77.90.31.19:4865): protocol: read tcp 188.166.225.170:39398->77.90.31.19:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231177167 (263h19m32.662257298s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: read tcp 188.166.225.170:58526->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231184571 (263h19m32.662264702s) [noise=true , address=45.91.116.12:4236 , piece_id=LCRQS2GOSHOSLZBGQTGB3Q3PA4VN7KXNW66TDZ5O424TPJWTQE3A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH, address:45.91.116.12:4236): protocol: read tcp 188.166.225.170:57166->45.91.116.12:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231189770 (263h19m32.662269901s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 188.166.225.170:54242->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231215850 (263h19m32.662295981s) [seg_pos={0 0}, upload=273764]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 19 16:45:38.459100173 (263h19m32.890180304s) [upload=273764]: Done uploading segments... Aug 19 16:45:38.459128316 (263h19m32.890208447s) [upload=273764]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.998953186s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X....X...........] duration: 6.086502412s FAIL (2025-08-19 16:57:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.086525826s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [..............X.X...X...X..............X......X.XX..........] duration: 6.27294003s FAIL (2025-08-19 16:57:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Aug 19 16:48:51.452491912 (263h22m45.883572043s) [upload=273834]: Starting upload Aug 19 16:48:51.452531028 (263h22m45.883611159s) [upload=273834]: Done starting upload Aug 19 16:48:51.452781006 (263h22m45.883861137s) [upload=273834]: Uploading segments... Aug 19 16:48:51.452853734 (263h22m45.883933866s) [upload=273834]: Got next segment. Inline: false Aug 19 16:48:51.452892670 (263h22m45.883972871s) [upload=273834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:48:51.452899813 (263h22m45.883980004s) [upload=273834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:48:51.962807465 (263h22m46.393887596s) [seg_pos={0 0}, upload=273834]: Begin upload segment... Aug 19 16:48:51.962852621 (263h22m46.393932743s) [seg_pos={0 0}, upload=273834]: Done begin upload segment. Aug 19 16:48:51.962875075 (263h22m46.393955206s) [noise=true , address=104.36.82.23:4470 , piece_id=5PDSSXU4HWGHBN62JJQBKHDQ6YYJVBWZZYSCJGKOPTZAZZUCAB6Q, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.962917746 (263h22m46.393997877s) [noise=true , address=66.23.193.175:4556 , piece_id=FBS7KZJF3HULYZZJDNLNOKMR5HSCR224IY7ZU4FUKZC5YMIIOAJA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963271022 (263h22m46.394351153s) [noise=true , address=45.91.116.9:4286 , piece_id=WYJJULBB3QVG2F46C52OCJWAMHVORGI7MDQS3CSYLF7REGAZQ3OQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963372556 (263h22m46.394452687s) [noise=true , address=156.146.43.225:4452 , piece_id=3T6HWTUPTX6A7BZULB4RSKGUO6GNDIHQGUP5W3EESJBWZGN7LLEQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963580684 (263h22m46.394660815s) [noise=true , address=104.36.82.35:4606 , piece_id=R37F22AIPFPV45K2PRWAJNONXQEDBJQPZGQKQ3LSHAQIBPKRXLHA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963653934 (263h22m46.394734066s) [noise=true , address=104.36.82.21:4478 , piece_id=XJWKXZ45KPRM3QUYXVIYU75M24GOCWGG5E3K4V62225XF3TGSS4Q, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963841874 (263h22m46.394922005s) [noise=true , address=79.127.201.215:4343 , piece_id=JF5HESY3ZARRZWZJHGTIG2BEZWWMXPHNJF5ALABQBAOVGJCHKI5A, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963910455 (263h22m46.394990586s) [noise=true , address=79.127.219.51:4211 , piece_id=OUB6QKVWCRJATMQFLAQ7UK3LJ4DILSLRAGFDP5YXROEXHZCPWGZA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964097293 (263h22m46.395177424s) [noise=true , address=104.36.82.60:4341 , piece_id=JCAHPCHE5IPHX35ZCDQ27I3DZ3R347HFRQWNBCNBXU6P3O7PTONA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964163640 (263h22m46.395243771s) [noise=true , address=77.90.31.18:4828 , piece_id=NFXT5LHIVX4G6DXMB4UWI37XCXLNHBRLVLXSJWTYAYN2JKCHRZQA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964400383 (263h22m46.395480514s) [noise=true , address=104.36.82.51:4052 , piece_id=7HLVSHJCDKI3TNKFETBUAY25HVIOPKAMSYQ6Q5BJ2NCJTQOTETTA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964624241 (263h22m46.395704372s) [noise=true , address=77.90.31.10:4529 , piece_id=HK5DLHGNULVF2RX7NNKFYN5JIFJB3LDEYXLXO3UN4D5OO4DJOW4A, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964832199 (263h22m46.39591233s) [noise=true , address=45.91.116.11:4130 , piece_id=5IZILPAZ4UKTFE766TWKT672JHDPWIQJI3TMOTASODK7NYG5BKPQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965048673 (263h22m46.396128804s) [noise=true , address=104.36.82.11:4127 , piece_id=6ACOOH24TTEMA4OQP6OFNHIABWDW3NDUVD44FSRIFLPTSVPHXOOA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965259626 (263h22m46.396339747s) [noise=true , address=77.90.31.15:4713 , piece_id=26S5ETN4OVNEL7IYYAIUBLP6UDS2QSSXCHD4H6YNHEQNY6FPOBOA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965452415 (263h22m46.396532546s) [noise=true , address=45.91.116.14:4209 , piece_id=5KAIVWUEWAG5NW6QGOAJLIMUADHZ6R4YTNAKTTXAGBRUEYXDAE7A, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965641637 (263h22m46.396721768s) [noise=true , address=45.91.116.7:4020 , piece_id=5NNTOPQ5BC7T2TLQ7NWSO3E6R4NDPB3JPXPMPRGG6I7YJU6QS4CQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965869794 (263h22m46.396949925s) [noise=true , address=77.90.31.17:4793 , piece_id=K3GCAOBIKBJV6Q3KOFXL5SEUFYE46MKJU5IAYNDA25IDCICTOFCQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:53.162668249 (263h22m47.59374838s) [upload=273834]: Next returned nil segment Aug 19 16:48:53.162683478 (263h22m47.593763599s) [upload=273834]: Waiting for error group managing segments... Aug 19 16:48:53.488132069 (263h22m47.91921223s) [noise=true , address=66.23.193.175:4556 , piece_id=FBS7KZJF3HULYZZJDNLNOKMR5HSCR224IY7ZU4FUKZC5YMIIOAJA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.887114541 (263h22m48.318194672s) [noise=true , address=79.127.219.51:4211 , piece_id=OUB6QKVWCRJATMQFLAQ7UK3LJ4DILSLRAGFDP5YXROEXHZCPWGZA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.925197211 (263h22m48.356277342s) [noise=true , address=156.146.43.225:4452 , piece_id=3T6HWTUPTX6A7BZULB4RSKGUO6GNDIHQGUP5W3EESJBWZGN7LLEQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.961937701 (263h22m48.393017832s) [noise=true , address=104.36.82.11:4127 , piece_id=6ACOOH24TTEMA4OQP6OFNHIABWDW3NDUVD44FSRIFLPTSVPHXOOA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.992310049 (263h22m48.42339018s) [noise=true , address=79.127.201.215:4343 , piece_id=JF5HESY3ZARRZWZJHGTIG2BEZWWMXPHNJF5ALABQBAOVGJCHKI5A, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.045243820 (263h22m48.476323961s) [noise=true , address=104.36.82.35:4606 , piece_id=R37F22AIPFPV45K2PRWAJNONXQEDBJQPZGQKQ3LSHAQIBPKRXLHA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.059875193 (263h22m48.490955324s) [noise=true , address=104.36.82.21:4478 , piece_id=XJWKXZ45KPRM3QUYXVIYU75M24GOCWGG5E3K4V62225XF3TGSS4Q, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.095629989 (263h22m48.52671012s) [noise=true , address=104.36.82.23:4470 , piece_id=5PDSSXU4HWGHBN62JJQBKHDQ6YYJVBWZZYSCJGKOPTZAZZUCAB6Q, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.103250568 (263h22m48.534330699s) [noise=true , address=104.36.82.51:4052 , piece_id=7HLVSHJCDKI3TNKFETBUAY25HVIOPKAMSYQ6Q5BJ2NCJTQOTETTA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.159963800 (263h22m48.591043921s) [noise=true , address=104.36.82.60:4341 , piece_id=JCAHPCHE5IPHX35ZCDQ27I3DZ3R347HFRQWNBCNBXU6P3O7PTONA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:49:37.214809078 (263h23m31.64588921s) [noise=true , address=77.90.31.17:4793 , piece_id=K3GCAOBIKBJV6Q3KOFXL5SEUFYE46MKJU5IAYNDA25IDCICTOFCQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:49:50.811468753 (263h23m45.242548894s) [noise=true , address=77.90.31.10:4529 , piece_id=HK5DLHGNULVF2RX7NNKFYN5JIFJB3LDEYXLXO3UN4D5OO4DJOW4A, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: read tcp 188.166.225.170:57426->77.90.31.10:4529: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811576489 (263h23m45.24265662s) [noise=true , address=45.91.116.14:4209 , piece_id=5KAIVWUEWAG5NW6QGOAJLIMUADHZ6R4YTNAKTTXAGBRUEYXDAE7A, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:12LWbAxsrDcyV31SxJNiVNYs7ERKAf6RTuKpE596UrYJAnXMndR, address:45.91.116.14:4209): protocol: rpc: dial tcp 45.91.116.14:4209: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811688974 (263h23m45.242769105s) [noise=true , address=45.91.116.9:4286 , piece_id=WYJJULBB3QVG2F46C52OCJWAMHVORGI7MDQS3CSYLF7REGAZQ3OQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:19DfbHig9tzF4VRHtSWkExwUS2bdZ8GFgKnAD9wvMBG7qaesNa, address:45.91.116.9:4286): protocol: read tcp 188.166.225.170:40032->45.91.116.9:4286: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811727057 (263h23m45.242807188s) [noise=true , address=45.91.116.11:4130 , piece_id=5IZILPAZ4UKTFE766TWKT672JHDPWIQJI3TMOTASODK7NYG5BKPQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:126ThYkfUAYf2JR8f5rkwyumcvmRArxKbpLUjhJukoc3FADHj2T, address:45.91.116.11:4130): protocol: read tcp 188.166.225.170:46860->45.91.116.11:4130: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811808934 (263h23m45.242889085s) [noise=true , address=45.91.116.7:4020 , piece_id=5NNTOPQ5BC7T2TLQ7NWSO3E6R4NDPB3JPXPMPRGG6I7YJU6QS4CQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:1MT5Nk9DRHLLfHi5sJnzWEjuL3aV2HnbS9Xebdw8vtBMjQmFnV, address:45.91.116.7:4020): protocol: read tcp 188.166.225.170:48966->45.91.116.7:4020: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811874559 (263h23m45.24295469s) [noise=true , address=77.90.31.18:4828 , piece_id=NFXT5LHIVX4G6DXMB4UWI37XCXLNHBRLVLXSJWTYAYN2JKCHRZQA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:1jwTDXPShXfNdfMx3UXi22gQphBpFre1MhRsHyjq6xRz4wwTr9, address:77.90.31.18:4828): protocol: read tcp 188.166.225.170:47612->77.90.31.18:4828: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811914556 (263h23m45.242994687s) [noise=true , address=77.90.31.15:4713 , piece_id=26S5ETN4OVNEL7IYYAIUBLP6UDS2QSSXCHD4H6YNHEQNY6FPOBOA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: read tcp 188.166.225.170:41506->77.90.31.15:4713: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811942720 (263h23m45.243022841s) [seg_pos={0 0}, upload=273834]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Aug 19 16:49:51.082762534 (263h23m45.513842665s) [upload=273834]: Done uploading segments... Aug 19 16:49:51.082792671 (263h23m45.513872803s) [upload=273834]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 6.27296599s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X...X............] duration: 6.651275881s FAIL (2025-08-19 16:57:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.651303675s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................X....XX............] duration: 6.259953171s FAIL (2025-08-19 16:57:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 19 16:49:12.801154313 (263h23m7.232234445s) [upload=273839]: Starting upload Aug 19 16:49:12.801215942 (263h23m7.232296073s) [upload=273839]: Done starting upload Aug 19 16:49:12.801330351 (263h23m7.232410482s) [upload=273839]: Uploading segments... Aug 19 16:49:12.801425372 (263h23m7.232505503s) [upload=273839]: Got next segment. Inline: false Aug 19 16:49:12.801455880 (263h23m7.232536012s) [upload=273839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:49:12.801461982 (263h23m7.232542113s) [upload=273839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:49:13.036708228 (263h23m7.467788359s) [seg_pos={0 0}, upload=273839]: Begin upload segment... Aug 19 16:49:13.036749197 (263h23m7.467829328s) [seg_pos={0 0}, upload=273839]: Done begin upload segment. Aug 19 16:49:13.036771971 (263h23m7.467852092s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.036780256 (263h23m7.467860437s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037135917 (263h23m7.468216048s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037326592 (263h23m7.468406723s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037403259 (263h23m7.46848339s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037667814 (263h23m7.468747945s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037761764 (263h23m7.468841896s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037943331 (263h23m7.469023462s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038070565 (263h23m7.469150696s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038168622 (263h23m7.469248753s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038300104 (263h23m7.469380235s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038396789 (263h23m7.46947692s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038622241 (263h23m7.469702362s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038679350 (263h23m7.469759481s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038854735 (263h23m7.469934856s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.039048045 (263h23m7.470128166s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:14.749064365 (263h23m9.180144496s) [upload=273839]: Next returned nil segment Aug 19 16:49:14.749117217 (263h23m9.180197348s) [upload=273839]: Waiting for error group managing segments... Aug 19 16:49:15.090317863 (263h23m9.521397994s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.132109007 (263h23m9.563189138s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.222090156 (263h23m9.653170287s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.328041780 (263h23m9.759121911s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.353998781 (263h23m9.785078912s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:36.709543777 (263h23m31.140623898s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:37.041062020 (263h23m31.472142141s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.183707791 (263h23m32.614787922s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.694075084 (263h23m33.125155215s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.716788163 (263h23m34.147868294s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.851890717 (263h23m34.282970848s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:40.293306092 (263h23m34.724386243s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:42.156929229 (263h23m36.58800936s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157075889 (263h23m36.58815601s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157126075 (263h23m36.588206206s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection Aug 19 16:49:42.157170951 (263h23m36.588251082s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: read tcp 188.166.225.170:34242->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:42.157215747 (263h23m36.588295878s) [seg_pos={0 0}, upload=273839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 19 16:49:42.385478364 (263h23m36.816558495s) [upload=273839]: Done uploading segments... Aug 19 16:49:42.385503091 (263h23m36.816583212s) [upload=273839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.259986475s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.205787611s FAIL (2025-08-19 16:56:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 263h18m33.75518017s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324344006 (263h18m33.755424127s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324633220 (263h18m33.755713351s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324851286 (263h18m33.755931417s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325068942 (263h18m33.756149063s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325283613 (263h18m33.756363744s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325499857 (263h18m33.756579998s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325759794 (263h18m33.756839915s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326059428 (263h18m33.757139549s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326277565 (263h18m33.757357696s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326470835 (263h18m33.757550966s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326673753 (263h18m33.757753884s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326867664 (263h18m33.757947795s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:40.851885389 (263h18m35.28296552s) [upload=273764]: Next returned nil segment Aug 19 16:44:40.851903334 (263h18m35.282983465s) [upload=273764]: Waiting for error group managing segments... Aug 19 16:44:41.244668372 (263h18m35.675748503s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.277280365 (263h18m35.708360496s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.328561665 (263h18m35.759641786s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.331747139 (263h18m35.76282727s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.357720320 (263h18m35.788800481s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.378190805 (263h18m35.809270936s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.486480333 (263h18m35.917560464s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:45:38.230570787 (263h19m32.661650918s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: expected piece hash; context deadline exceeded; EOF Aug 19 16:45:38.230979869 (263h19m32.66206s) [noise=true , address=77.90.31.18:4840 , piece_id=JWOETSFSZ3HELLZNLEASYNGW5OR64TMQVOFYYJAJVTHWCBHVTSLQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: read tcp 188.166.225.170:39946->77.90.31.18:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231017812 (263h19m32.662097943s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 188.166.225.170:44164->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231063489 (263h19m32.66214362s) [noise=true , address=45.91.116.14:4203 , piece_id=WRLBES3FI5EA5AQLZXJYGDROP3KBG3E2F22R4YD6A5KKI44PLV2Q, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 188.166.225.170:51738->45.91.116.14:4203: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231090952 (263h19m32.662171083s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: read tcp 188.166.225.170:57158->77.90.31.15:4707: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231110719 (263h19m32.66219085s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:50784->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231140165 (263h19m32.662220296s) [noise=true , address=77.90.31.16:4749 , piece_id=XCPXVBYLQH345FPIW4KDNLBFUCDDEW2H4B25WIXGOO5FVMAIVKPQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: read tcp 188.166.225.170:51668->77.90.31.16:4749: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231144814 (263h19m32.662224945s) [noise=true , address=77.90.31.19:4865 , piece_id=B3HUFIJZKVZJH4OBBO2DD7S3I7HX4LO2IQ74XJPO4DTW4QER35NA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1bguBghQonV5sTHaWteyUEnpp5KNzBKbvBpbvrUNFDCveHN77X, address:77.90.31.19:4865): protocol: read tcp 188.166.225.170:39398->77.90.31.19:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231177167 (263h19m32.662257298s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: read tcp 188.166.225.170:58526->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231184571 (263h19m32.662264702s) [noise=true , address=45.91.116.12:4236 , piece_id=LCRQS2GOSHOSLZBGQTGB3Q3PA4VN7KXNW66TDZ5O424TPJWTQE3A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH, address:45.91.116.12:4236): protocol: read tcp 188.166.225.170:57166->45.91.116.12:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231189770 (263h19m32.662269901s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 188.166.225.170:54242->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231215850 (263h19m32.662295981s) [seg_pos={0 0}, upload=273764]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 19 16:45:38.459100173 (263h19m32.890180304s) [upload=273764]: Done uploading segments... Aug 19 16:45:38.459128316 (263h19m32.890208447s) [upload=273764]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 6.205808139s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X....X..........] duration: 5.98164453s FAIL (2025-08-19 16:56:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.981703163s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [...............X.X...X...X..............X......X.XX.........] duration: 5.900336014s FAIL (2025-08-19 16:56:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Aug 19 16:48:51.452491912 (263h22m45.883572043s) [upload=273834]: Starting upload Aug 19 16:48:51.452531028 (263h22m45.883611159s) [upload=273834]: Done starting upload Aug 19 16:48:51.452781006 (263h22m45.883861137s) [upload=273834]: Uploading segments... Aug 19 16:48:51.452853734 (263h22m45.883933866s) [upload=273834]: Got next segment. Inline: false Aug 19 16:48:51.452892670 (263h22m45.883972871s) [upload=273834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:48:51.452899813 (263h22m45.883980004s) [upload=273834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:48:51.962807465 (263h22m46.393887596s) [seg_pos={0 0}, upload=273834]: Begin upload segment... Aug 19 16:48:51.962852621 (263h22m46.393932743s) [seg_pos={0 0}, upload=273834]: Done begin upload segment. Aug 19 16:48:51.962875075 (263h22m46.393955206s) [noise=true , address=104.36.82.23:4470 , piece_id=5PDSSXU4HWGHBN62JJQBKHDQ6YYJVBWZZYSCJGKOPTZAZZUCAB6Q, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.962917746 (263h22m46.393997877s) [noise=true , address=66.23.193.175:4556 , piece_id=FBS7KZJF3HULYZZJDNLNOKMR5HSCR224IY7ZU4FUKZC5YMIIOAJA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963271022 (263h22m46.394351153s) [noise=true , address=45.91.116.9:4286 , piece_id=WYJJULBB3QVG2F46C52OCJWAMHVORGI7MDQS3CSYLF7REGAZQ3OQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963372556 (263h22m46.394452687s) [noise=true , address=156.146.43.225:4452 , piece_id=3T6HWTUPTX6A7BZULB4RSKGUO6GNDIHQGUP5W3EESJBWZGN7LLEQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963580684 (263h22m46.394660815s) [noise=true , address=104.36.82.35:4606 , piece_id=R37F22AIPFPV45K2PRWAJNONXQEDBJQPZGQKQ3LSHAQIBPKRXLHA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963653934 (263h22m46.394734066s) [noise=true , address=104.36.82.21:4478 , piece_id=XJWKXZ45KPRM3QUYXVIYU75M24GOCWGG5E3K4V62225XF3TGSS4Q, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963841874 (263h22m46.394922005s) [noise=true , address=79.127.201.215:4343 , piece_id=JF5HESY3ZARRZWZJHGTIG2BEZWWMXPHNJF5ALABQBAOVGJCHKI5A, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963910455 (263h22m46.394990586s) [noise=true , address=79.127.219.51:4211 , piece_id=OUB6QKVWCRJATMQFLAQ7UK3LJ4DILSLRAGFDP5YXROEXHZCPWGZA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964097293 (263h22m46.395177424s) [noise=true , address=104.36.82.60:4341 , piece_id=JCAHPCHE5IPHX35ZCDQ27I3DZ3R347HFRQWNBCNBXU6P3O7PTONA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964163640 (263h22m46.395243771s) [noise=true , address=77.90.31.18:4828 , piece_id=NFXT5LHIVX4G6DXMB4UWI37XCXLNHBRLVLXSJWTYAYN2JKCHRZQA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964400383 (263h22m46.395480514s) [noise=true , address=104.36.82.51:4052 , piece_id=7HLVSHJCDKI3TNKFETBUAY25HVIOPKAMSYQ6Q5BJ2NCJTQOTETTA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964624241 (263h22m46.395704372s) [noise=true , address=77.90.31.10:4529 , piece_id=HK5DLHGNULVF2RX7NNKFYN5JIFJB3LDEYXLXO3UN4D5OO4DJOW4A, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964832199 (263h22m46.39591233s) [noise=true , address=45.91.116.11:4130 , piece_id=5IZILPAZ4UKTFE766TWKT672JHDPWIQJI3TMOTASODK7NYG5BKPQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965048673 (263h22m46.396128804s) [noise=true , address=104.36.82.11:4127 , piece_id=6ACOOH24TTEMA4OQP6OFNHIABWDW3NDUVD44FSRIFLPTSVPHXOOA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965259626 (263h22m46.396339747s) [noise=true , address=77.90.31.15:4713 , piece_id=26S5ETN4OVNEL7IYYAIUBLP6UDS2QSSXCHD4H6YNHEQNY6FPOBOA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965452415 (263h22m46.396532546s) [noise=true , address=45.91.116.14:4209 , piece_id=5KAIVWUEWAG5NW6QGOAJLIMUADHZ6R4YTNAKTTXAGBRUEYXDAE7A, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965641637 (263h22m46.396721768s) [noise=true , address=45.91.116.7:4020 , piece_id=5NNTOPQ5BC7T2TLQ7NWSO3E6R4NDPB3JPXPMPRGG6I7YJU6QS4CQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965869794 (263h22m46.396949925s) [noise=true , address=77.90.31.17:4793 , piece_id=K3GCAOBIKBJV6Q3KOFXL5SEUFYE46MKJU5IAYNDA25IDCICTOFCQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:53.162668249 (263h22m47.59374838s) [upload=273834]: Next returned nil segment Aug 19 16:48:53.162683478 (263h22m47.593763599s) [upload=273834]: Waiting for error group managing segments... Aug 19 16:48:53.488132069 (263h22m47.91921223s) [noise=true , address=66.23.193.175:4556 , piece_id=FBS7KZJF3HULYZZJDNLNOKMR5HSCR224IY7ZU4FUKZC5YMIIOAJA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.887114541 (263h22m48.318194672s) [noise=true , address=79.127.219.51:4211 , piece_id=OUB6QKVWCRJATMQFLAQ7UK3LJ4DILSLRAGFDP5YXROEXHZCPWGZA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.925197211 (263h22m48.356277342s) [noise=true , address=156.146.43.225:4452 , piece_id=3T6HWTUPTX6A7BZULB4RSKGUO6GNDIHQGUP5W3EESJBWZGN7LLEQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.961937701 (263h22m48.393017832s) [noise=true , address=104.36.82.11:4127 , piece_id=6ACOOH24TTEMA4OQP6OFNHIABWDW3NDUVD44FSRIFLPTSVPHXOOA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.992310049 (263h22m48.42339018s) [noise=true , address=79.127.201.215:4343 , piece_id=JF5HESY3ZARRZWZJHGTIG2BEZWWMXPHNJF5ALABQBAOVGJCHKI5A, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.045243820 (263h22m48.476323961s) [noise=true , address=104.36.82.35:4606 , piece_id=R37F22AIPFPV45K2PRWAJNONXQEDBJQPZGQKQ3LSHAQIBPKRXLHA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.059875193 (263h22m48.490955324s) [noise=true , address=104.36.82.21:4478 , piece_id=XJWKXZ45KPRM3QUYXVIYU75M24GOCWGG5E3K4V62225XF3TGSS4Q, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.095629989 (263h22m48.52671012s) [noise=true , address=104.36.82.23:4470 , piece_id=5PDSSXU4HWGHBN62JJQBKHDQ6YYJVBWZZYSCJGKOPTZAZZUCAB6Q, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.103250568 (263h22m48.534330699s) [noise=true , address=104.36.82.51:4052 , piece_id=7HLVSHJCDKI3TNKFETBUAY25HVIOPKAMSYQ6Q5BJ2NCJTQOTETTA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.159963800 (263h22m48.591043921s) [noise=true , address=104.36.82.60:4341 , piece_id=JCAHPCHE5IPHX35ZCDQ27I3DZ3R347HFRQWNBCNBXU6P3O7PTONA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:49:37.214809078 (263h23m31.64588921s) [noise=true , address=77.90.31.17:4793 , piece_id=K3GCAOBIKBJV6Q3KOFXL5SEUFYE46MKJU5IAYNDA25IDCICTOFCQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:49:50.811468753 (263h23m45.242548894s) [noise=true , address=77.90.31.10:4529 , piece_id=HK5DLHGNULVF2RX7NNKFYN5JIFJB3LDEYXLXO3UN4D5OO4DJOW4A, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: read tcp 188.166.225.170:57426->77.90.31.10:4529: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811576489 (263h23m45.24265662s) [noise=true , address=45.91.116.14:4209 , piece_id=5KAIVWUEWAG5NW6QGOAJLIMUADHZ6R4YTNAKTTXAGBRUEYXDAE7A, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:12LWbAxsrDcyV31SxJNiVNYs7ERKAf6RTuKpE596UrYJAnXMndR, address:45.91.116.14:4209): protocol: rpc: dial tcp 45.91.116.14:4209: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811688974 (263h23m45.242769105s) [noise=true , address=45.91.116.9:4286 , piece_id=WYJJULBB3QVG2F46C52OCJWAMHVORGI7MDQS3CSYLF7REGAZQ3OQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:19DfbHig9tzF4VRHtSWkExwUS2bdZ8GFgKnAD9wvMBG7qaesNa, address:45.91.116.9:4286): protocol: read tcp 188.166.225.170:40032->45.91.116.9:4286: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811727057 (263h23m45.242807188s) [noise=true , address=45.91.116.11:4130 , piece_id=5IZILPAZ4UKTFE766TWKT672JHDPWIQJI3TMOTASODK7NYG5BKPQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:126ThYkfUAYf2JR8f5rkwyumcvmRArxKbpLUjhJukoc3FADHj2T, address:45.91.116.11:4130): protocol: read tcp 188.166.225.170:46860->45.91.116.11:4130: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811808934 (263h23m45.242889085s) [noise=true , address=45.91.116.7:4020 , piece_id=5NNTOPQ5BC7T2TLQ7NWSO3E6R4NDPB3JPXPMPRGG6I7YJU6QS4CQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:1MT5Nk9DRHLLfHi5sJnzWEjuL3aV2HnbS9Xebdw8vtBMjQmFnV, address:45.91.116.7:4020): protocol: read tcp 188.166.225.170:48966->45.91.116.7:4020: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811874559 (263h23m45.24295469s) [noise=true , address=77.90.31.18:4828 , piece_id=NFXT5LHIVX4G6DXMB4UWI37XCXLNHBRLVLXSJWTYAYN2JKCHRZQA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:1jwTDXPShXfNdfMx3UXi22gQphBpFre1MhRsHyjq6xRz4wwTr9, address:77.90.31.18:4828): protocol: read tcp 188.166.225.170:47612->77.90.31.18:4828: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811914556 (263h23m45.242994687s) [noise=true , address=77.90.31.15:4713 , piece_id=26S5ETN4OVNEL7IYYAIUBLP6UDS2QSSXCHD4H6YNHEQNY6FPOBOA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: read tcp 188.166.225.170:41506->77.90.31.15:4713: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811942720 (263h23m45.243022841s) [seg_pos={0 0}, upload=273834]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Aug 19 16:49:51.082762534 (263h23m45.513842665s) [upload=273834]: Done uploading segments... Aug 19 16:49:51.082792671 (263h23m45.513872803s) [upload=273834]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.900383054s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............................................X...X...........] duration: 6.574951647s FAIL (2025-08-19 16:56:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.575346773s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [............X...............X...............................] duration: 2.269175923s FAIL (2025-08-19 16:56:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after be5a5f4d20475bc6e6586ca4cb41bd260785c25704d0f801265da5a966efbd34 != before fab796bcbc8cefce4dfe0fedb7e23ecb2129705e75b1b66c720b4c58f0a82c58 ======LOGS====== Aug 19 16:31:34.462024854 (263h5m28.893104995s) [upload=273541]: Starting upload Aug 19 16:31:34.462073225 (263h5m28.893153356s) [upload=273541]: Done starting upload Aug 19 16:31:34.462124303 (263h5m28.893204434s) [upload=273541]: Uploading segments... Aug 19 16:31:34.462159490 (263h5m28.893239621s) [upload=273541]: Got next segment. Inline: true Aug 19 16:31:34.462163108 (263h5m28.893243239s) [upload=273541]: Waiting for error group managing segments... Aug 19 16:31:34.462196912 (263h5m28.893277044s) [upload=273541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:31:34.462200970 (263h5m28.893281101s) [upload=273541]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:31:34.462203415 (263h5m28.893283546s) [upload=273541]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:31:34.760143563 (263h5m29.191223694s) [upload=273541]: Done uploading segments... Aug 19 16:31:34.760161496 (263h5m29.191241628s) [upload=273541]: Upload finished. err: <nil> Duration:* 2.269199239s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................................X....XX...........] duration: 6.331695843s FAIL (2025-08-19 16:56:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 19 16:49:12.801154313 (263h23m7.232234445s) [upload=273839]: Starting upload Aug 19 16:49:12.801215942 (263h23m7.232296073s) [upload=273839]: Done starting upload Aug 19 16:49:12.801330351 (263h23m7.232410482s) [upload=273839]: Uploading segments... Aug 19 16:49:12.801425372 (263h23m7.232505503s) [upload=273839]: Got next segment. Inline: false Aug 19 16:49:12.801455880 (263h23m7.232536012s) [upload=273839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:49:12.801461982 (263h23m7.232542113s) [upload=273839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:49:13.036708228 (263h23m7.467788359s) [seg_pos={0 0}, upload=273839]: Begin upload segment... Aug 19 16:49:13.036749197 (263h23m7.467829328s) [seg_pos={0 0}, upload=273839]: Done begin upload segment. Aug 19 16:49:13.036771971 (263h23m7.467852092s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.036780256 (263h23m7.467860437s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037135917 (263h23m7.468216048s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037326592 (263h23m7.468406723s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037403259 (263h23m7.46848339s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037667814 (263h23m7.468747945s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037761764 (263h23m7.468841896s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037943331 (263h23m7.469023462s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038070565 (263h23m7.469150696s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038168622 (263h23m7.469248753s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038300104 (263h23m7.469380235s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038396789 (263h23m7.46947692s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038622241 (263h23m7.469702362s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038679350 (263h23m7.469759481s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038854735 (263h23m7.469934856s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.039048045 (263h23m7.470128166s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:14.749064365 (263h23m9.180144496s) [upload=273839]: Next returned nil segment Aug 19 16:49:14.749117217 (263h23m9.180197348s) [upload=273839]: Waiting for error group managing segments... Aug 19 16:49:15.090317863 (263h23m9.521397994s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.132109007 (263h23m9.563189138s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.222090156 (263h23m9.653170287s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.328041780 (263h23m9.759121911s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.353998781 (263h23m9.785078912s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:36.709543777 (263h23m31.140623898s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:37.041062020 (263h23m31.472142141s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.183707791 (263h23m32.614787922s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.694075084 (263h23m33.125155215s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.716788163 (263h23m34.147868294s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.851890717 (263h23m34.282970848s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:40.293306092 (263h23m34.724386243s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:42.156929229 (263h23m36.58800936s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157075889 (263h23m36.58815601s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157126075 (263h23m36.588206206s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection Aug 19 16:49:42.157170951 (263h23m36.588251082s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: read tcp 188.166.225.170:34242->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:42.157215747 (263h23m36.588295878s) [seg_pos={0 0}, upload=273839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 19 16:49:42.385478364 (263h23m36.816558495s) [upload=273839]: Done uploading segments... Aug 19 16:49:42.385503091 (263h23m36.816583212s) [upload=273839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.331738865s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X....X.........] duration: 5.906127345s FAIL (2025-08-19 16:56:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.906152503s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X...X..........] duration: 6.506871974s FAIL (2025-08-19 16:56:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.506891582s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 8.53945317s FAIL (2025-08-19 16:55:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 263h18m33.75518017s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324344006 (263h18m33.755424127s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324633220 (263h18m33.755713351s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324851286 (263h18m33.755931417s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325068942 (263h18m33.756149063s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325283613 (263h18m33.756363744s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325499857 (263h18m33.756579998s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325759794 (263h18m33.756839915s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326059428 (263h18m33.757139549s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326277565 (263h18m33.757357696s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326470835 (263h18m33.757550966s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326673753 (263h18m33.757753884s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326867664 (263h18m33.757947795s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:40.851885389 (263h18m35.28296552s) [upload=273764]: Next returned nil segment Aug 19 16:44:40.851903334 (263h18m35.282983465s) [upload=273764]: Waiting for error group managing segments... Aug 19 16:44:41.244668372 (263h18m35.675748503s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.277280365 (263h18m35.708360496s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.328561665 (263h18m35.759641786s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.331747139 (263h18m35.76282727s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.357720320 (263h18m35.788800481s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.378190805 (263h18m35.809270936s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.486480333 (263h18m35.917560464s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:45:38.230570787 (263h19m32.661650918s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: expected piece hash; context deadline exceeded; EOF Aug 19 16:45:38.230979869 (263h19m32.66206s) [noise=true , address=77.90.31.18:4840 , piece_id=JWOETSFSZ3HELLZNLEASYNGW5OR64TMQVOFYYJAJVTHWCBHVTSLQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: read tcp 188.166.225.170:39946->77.90.31.18:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231017812 (263h19m32.662097943s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 188.166.225.170:44164->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231063489 (263h19m32.66214362s) [noise=true , address=45.91.116.14:4203 , piece_id=WRLBES3FI5EA5AQLZXJYGDROP3KBG3E2F22R4YD6A5KKI44PLV2Q, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 188.166.225.170:51738->45.91.116.14:4203: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231090952 (263h19m32.662171083s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: read tcp 188.166.225.170:57158->77.90.31.15:4707: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231110719 (263h19m32.66219085s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:50784->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231140165 (263h19m32.662220296s) [noise=true , address=77.90.31.16:4749 , piece_id=XCPXVBYLQH345FPIW4KDNLBFUCDDEW2H4B25WIXGOO5FVMAIVKPQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: read tcp 188.166.225.170:51668->77.90.31.16:4749: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231144814 (263h19m32.662224945s) [noise=true , address=77.90.31.19:4865 , piece_id=B3HUFIJZKVZJH4OBBO2DD7S3I7HX4LO2IQ74XJPO4DTW4QER35NA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1bguBghQonV5sTHaWteyUEnpp5KNzBKbvBpbvrUNFDCveHN77X, address:77.90.31.19:4865): protocol: read tcp 188.166.225.170:39398->77.90.31.19:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231177167 (263h19m32.662257298s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: read tcp 188.166.225.170:58526->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231184571 (263h19m32.662264702s) [noise=true , address=45.91.116.12:4236 , piece_id=LCRQS2GOSHOSLZBGQTGB3Q3PA4VN7KXNW66TDZ5O424TPJWTQE3A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH, address:45.91.116.12:4236): protocol: read tcp 188.166.225.170:57166->45.91.116.12:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231189770 (263h19m32.662269901s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 188.166.225.170:54242->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231215850 (263h19m32.662295981s) [seg_pos={0 0}, upload=273764]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 19 16:45:38.459100173 (263h19m32.890180304s) [upload=273764]: Done uploading segments... Aug 19 16:45:38.459128316 (263h19m32.890208447s) [upload=273764]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 8.539481474s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.............X...............X..............................] duration: 2.209651772s FAIL (2025-08-19 16:55:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after be5a5f4d20475bc6e6586ca4cb41bd260785c25704d0f801265da5a966efbd34 != before fab796bcbc8cefce4dfe0fedb7e23ecb2129705e75b1b66c720b4c58f0a82c58 ======LOGS====== Aug 19 16:31:34.462024854 (263h5m28.893104995s) [upload=273541]: Starting upload Aug 19 16:31:34.462073225 (263h5m28.893153356s) [upload=273541]: Done starting upload Aug 19 16:31:34.462124303 (263h5m28.893204434s) [upload=273541]: Uploading segments... Aug 19 16:31:34.462159490 (263h5m28.893239621s) [upload=273541]: Got next segment. Inline: true Aug 19 16:31:34.462163108 (263h5m28.893243239s) [upload=273541]: Waiting for error group managing segments... Aug 19 16:31:34.462196912 (263h5m28.893277044s) [upload=273541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:31:34.462200970 (263h5m28.893281101s) [upload=273541]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:31:34.462203415 (263h5m28.893283546s) [upload=273541]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:31:34.760143563 (263h5m29.191223694s) [upload=273541]: Done uploading segments... Aug 19 16:31:34.760161496 (263h5m29.191241628s) [upload=273541]: Upload finished. err: <nil> Duration:* 2.209686829s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................................X....XX..........] duration: 6.491755008s FAIL (2025-08-19 16:55:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 19 16:49:12.801154313 (263h23m7.232234445s) [upload=273839]: Starting upload Aug 19 16:49:12.801215942 (263h23m7.232296073s) [upload=273839]: Done starting upload Aug 19 16:49:12.801330351 (263h23m7.232410482s) [upload=273839]: Uploading segments... Aug 19 16:49:12.801425372 (263h23m7.232505503s) [upload=273839]: Got next segment. Inline: false Aug 19 16:49:12.801455880 (263h23m7.232536012s) [upload=273839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:49:12.801461982 (263h23m7.232542113s) [upload=273839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:49:13.036708228 (263h23m7.467788359s) [seg_pos={0 0}, upload=273839]: Begin upload segment... Aug 19 16:49:13.036749197 (263h23m7.467829328s) [seg_pos={0 0}, upload=273839]: Done begin upload segment. Aug 19 16:49:13.036771971 (263h23m7.467852092s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.036780256 (263h23m7.467860437s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037135917 (263h23m7.468216048s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037326592 (263h23m7.468406723s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037403259 (263h23m7.46848339s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037667814 (263h23m7.468747945s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037761764 (263h23m7.468841896s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037943331 (263h23m7.469023462s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038070565 (263h23m7.469150696s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038168622 (263h23m7.469248753s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038300104 (263h23m7.469380235s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038396789 (263h23m7.46947692s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038622241 (263h23m7.469702362s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038679350 (263h23m7.469759481s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038854735 (263h23m7.469934856s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.039048045 (263h23m7.470128166s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:14.749064365 (263h23m9.180144496s) [upload=273839]: Next returned nil segment Aug 19 16:49:14.749117217 (263h23m9.180197348s) [upload=273839]: Waiting for error group managing segments... Aug 19 16:49:15.090317863 (263h23m9.521397994s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.132109007 (263h23m9.563189138s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.222090156 (263h23m9.653170287s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.328041780 (263h23m9.759121911s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.353998781 (263h23m9.785078912s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:36.709543777 (263h23m31.140623898s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:37.041062020 (263h23m31.472142141s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.183707791 (263h23m32.614787922s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.694075084 (263h23m33.125155215s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.716788163 (263h23m34.147868294s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.851890717 (263h23m34.282970848s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:40.293306092 (263h23m34.724386243s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:42.156929229 (263h23m36.58800936s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157075889 (263h23m36.58815601s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157126075 (263h23m36.588206206s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection Aug 19 16:49:42.157170951 (263h23m36.588251082s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: read tcp 188.166.225.170:34242->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:42.157215747 (263h23m36.588295878s) [seg_pos={0 0}, upload=273839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 19 16:49:42.385478364 (263h23m36.816558495s) [upload=273839]: Done uploading segments... Aug 19 16:49:42.385503091 (263h23m36.816583212s) [upload=273839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.491783012s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [................X.X...X...X..............X......X.XX........] duration: 5.998767422s FAIL (2025-08-19 16:55:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Aug 19 16:48:51.452491912 (263h22m45.883572043s) [upload=273834]: Starting upload Aug 19 16:48:51.452531028 (263h22m45.883611159s) [upload=273834]: Done starting upload Aug 19 16:48:51.452781006 (263h22m45.883861137s) [upload=273834]: Uploading segments... Aug 19 16:48:51.452853734 (263h22m45.883933866s) [upload=273834]: Got next segment. Inline: false Aug 19 16:48:51.452892670 (263h22m45.883972871s) [upload=273834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:48:51.452899813 (263h22m45.883980004s) [upload=273834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:48:51.962807465 (263h22m46.393887596s) [seg_pos={0 0}, upload=273834]: Begin upload segment... Aug 19 16:48:51.962852621 (263h22m46.393932743s) [seg_pos={0 0}, upload=273834]: Done begin upload segment. Aug 19 16:48:51.962875075 (263h22m46.393955206s) [noise=true , address=104.36.82.23:4470 , piece_id=5PDSSXU4HWGHBN62JJQBKHDQ6YYJVBWZZYSCJGKOPTZAZZUCAB6Q, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.962917746 (263h22m46.393997877s) [noise=true , address=66.23.193.175:4556 , piece_id=FBS7KZJF3HULYZZJDNLNOKMR5HSCR224IY7ZU4FUKZC5YMIIOAJA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963271022 (263h22m46.394351153s) [noise=true , address=45.91.116.9:4286 , piece_id=WYJJULBB3QVG2F46C52OCJWAMHVORGI7MDQS3CSYLF7REGAZQ3OQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963372556 (263h22m46.394452687s) [noise=true , address=156.146.43.225:4452 , piece_id=3T6HWTUPTX6A7BZULB4RSKGUO6GNDIHQGUP5W3EESJBWZGN7LLEQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963580684 (263h22m46.394660815s) [noise=true , address=104.36.82.35:4606 , piece_id=R37F22AIPFPV45K2PRWAJNONXQEDBJQPZGQKQ3LSHAQIBPKRXLHA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963653934 (263h22m46.394734066s) [noise=true , address=104.36.82.21:4478 , piece_id=XJWKXZ45KPRM3QUYXVIYU75M24GOCWGG5E3K4V62225XF3TGSS4Q, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963841874 (263h22m46.394922005s) [noise=true , address=79.127.201.215:4343 , piece_id=JF5HESY3ZARRZWZJHGTIG2BEZWWMXPHNJF5ALABQBAOVGJCHKI5A, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963910455 (263h22m46.394990586s) [noise=true , address=79.127.219.51:4211 , piece_id=OUB6QKVWCRJATMQFLAQ7UK3LJ4DILSLRAGFDP5YXROEXHZCPWGZA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964097293 (263h22m46.395177424s) [noise=true , address=104.36.82.60:4341 , piece_id=JCAHPCHE5IPHX35ZCDQ27I3DZ3R347HFRQWNBCNBXU6P3O7PTONA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964163640 (263h22m46.395243771s) [noise=true , address=77.90.31.18:4828 , piece_id=NFXT5LHIVX4G6DXMB4UWI37XCXLNHBRLVLXSJWTYAYN2JKCHRZQA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964400383 (263h22m46.395480514s) [noise=true , address=104.36.82.51:4052 , piece_id=7HLVSHJCDKI3TNKFETBUAY25HVIOPKAMSYQ6Q5BJ2NCJTQOTETTA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964624241 (263h22m46.395704372s) [noise=true , address=77.90.31.10:4529 , piece_id=HK5DLHGNULVF2RX7NNKFYN5JIFJB3LDEYXLXO3UN4D5OO4DJOW4A, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964832199 (263h22m46.39591233s) [noise=true , address=45.91.116.11:4130 , piece_id=5IZILPAZ4UKTFE766TWKT672JHDPWIQJI3TMOTASODK7NYG5BKPQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965048673 (263h22m46.396128804s) [noise=true , address=104.36.82.11:4127 , piece_id=6ACOOH24TTEMA4OQP6OFNHIABWDW3NDUVD44FSRIFLPTSVPHXOOA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965259626 (263h22m46.396339747s) [noise=true , address=77.90.31.15:4713 , piece_id=26S5ETN4OVNEL7IYYAIUBLP6UDS2QSSXCHD4H6YNHEQNY6FPOBOA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965452415 (263h22m46.396532546s) [noise=true , address=45.91.116.14:4209 , piece_id=5KAIVWUEWAG5NW6QGOAJLIMUADHZ6R4YTNAKTTXAGBRUEYXDAE7A, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965641637 (263h22m46.396721768s) [noise=true , address=45.91.116.7:4020 , piece_id=5NNTOPQ5BC7T2TLQ7NWSO3E6R4NDPB3JPXPMPRGG6I7YJU6QS4CQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965869794 (263h22m46.396949925s) [noise=true , address=77.90.31.17:4793 , piece_id=K3GCAOBIKBJV6Q3KOFXL5SEUFYE46MKJU5IAYNDA25IDCICTOFCQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:53.162668249 (263h22m47.59374838s) [upload=273834]: Next returned nil segment Aug 19 16:48:53.162683478 (263h22m47.593763599s) [upload=273834]: Waiting for error group managing segments... Aug 19 16:48:53.488132069 (263h22m47.91921223s) [noise=true , address=66.23.193.175:4556 , piece_id=FBS7KZJF3HULYZZJDNLNOKMR5HSCR224IY7ZU4FUKZC5YMIIOAJA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.887114541 (263h22m48.318194672s) [noise=true , address=79.127.219.51:4211 , piece_id=OUB6QKVWCRJATMQFLAQ7UK3LJ4DILSLRAGFDP5YXROEXHZCPWGZA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.925197211 (263h22m48.356277342s) [noise=true , address=156.146.43.225:4452 , piece_id=3T6HWTUPTX6A7BZULB4RSKGUO6GNDIHQGUP5W3EESJBWZGN7LLEQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.961937701 (263h22m48.393017832s) [noise=true , address=104.36.82.11:4127 , piece_id=6ACOOH24TTEMA4OQP6OFNHIABWDW3NDUVD44FSRIFLPTSVPHXOOA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.992310049 (263h22m48.42339018s) [noise=true , address=79.127.201.215:4343 , piece_id=JF5HESY3ZARRZWZJHGTIG2BEZWWMXPHNJF5ALABQBAOVGJCHKI5A, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.045243820 (263h22m48.476323961s) [noise=true , address=104.36.82.35:4606 , piece_id=R37F22AIPFPV45K2PRWAJNONXQEDBJQPZGQKQ3LSHAQIBPKRXLHA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.059875193 (263h22m48.490955324s) [noise=true , address=104.36.82.21:4478 , piece_id=XJWKXZ45KPRM3QUYXVIYU75M24GOCWGG5E3K4V62225XF3TGSS4Q, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.095629989 (263h22m48.52671012s) [noise=true , address=104.36.82.23:4470 , piece_id=5PDSSXU4HWGHBN62JJQBKHDQ6YYJVBWZZYSCJGKOPTZAZZUCAB6Q, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.103250568 (263h22m48.534330699s) [noise=true , address=104.36.82.51:4052 , piece_id=7HLVSHJCDKI3TNKFETBUAY25HVIOPKAMSYQ6Q5BJ2NCJTQOTETTA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.159963800 (263h22m48.591043921s) [noise=true , address=104.36.82.60:4341 , piece_id=JCAHPCHE5IPHX35ZCDQ27I3DZ3R347HFRQWNBCNBXU6P3O7PTONA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:49:37.214809078 (263h23m31.64588921s) [noise=true , address=77.90.31.17:4793 , piece_id=K3GCAOBIKBJV6Q3KOFXL5SEUFYE46MKJU5IAYNDA25IDCICTOFCQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:49:50.811468753 (263h23m45.242548894s) [noise=true , address=77.90.31.10:4529 , piece_id=HK5DLHGNULVF2RX7NNKFYN5JIFJB3LDEYXLXO3UN4D5OO4DJOW4A, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: read tcp 188.166.225.170:57426->77.90.31.10:4529: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811576489 (263h23m45.24265662s) [noise=true , address=45.91.116.14:4209 , piece_id=5KAIVWUEWAG5NW6QGOAJLIMUADHZ6R4YTNAKTTXAGBRUEYXDAE7A, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:12LWbAxsrDcyV31SxJNiVNYs7ERKAf6RTuKpE596UrYJAnXMndR, address:45.91.116.14:4209): protocol: rpc: dial tcp 45.91.116.14:4209: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811688974 (263h23m45.242769105s) [noise=true , address=45.91.116.9:4286 , piece_id=WYJJULBB3QVG2F46C52OCJWAMHVORGI7MDQS3CSYLF7REGAZQ3OQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:19DfbHig9tzF4VRHtSWkExwUS2bdZ8GFgKnAD9wvMBG7qaesNa, address:45.91.116.9:4286): protocol: read tcp 188.166.225.170:40032->45.91.116.9:4286: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811727057 (263h23m45.242807188s) [noise=true , address=45.91.116.11:4130 , piece_id=5IZILPAZ4UKTFE766TWKT672JHDPWIQJI3TMOTASODK7NYG5BKPQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:126ThYkfUAYf2JR8f5rkwyumcvmRArxKbpLUjhJukoc3FADHj2T, address:45.91.116.11:4130): protocol: read tcp 188.166.225.170:46860->45.91.116.11:4130: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811808934 (263h23m45.242889085s) [noise=true , address=45.91.116.7:4020 , piece_id=5NNTOPQ5BC7T2TLQ7NWSO3E6R4NDPB3JPXPMPRGG6I7YJU6QS4CQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:1MT5Nk9DRHLLfHi5sJnzWEjuL3aV2HnbS9Xebdw8vtBMjQmFnV, address:45.91.116.7:4020): protocol: read tcp 188.166.225.170:48966->45.91.116.7:4020: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811874559 (263h23m45.24295469s) [noise=true , address=77.90.31.18:4828 , piece_id=NFXT5LHIVX4G6DXMB4UWI37XCXLNHBRLVLXSJWTYAYN2JKCHRZQA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:1jwTDXPShXfNdfMx3UXi22gQphBpFre1MhRsHyjq6xRz4wwTr9, address:77.90.31.18:4828): protocol: read tcp 188.166.225.170:47612->77.90.31.18:4828: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811914556 (263h23m45.242994687s) [noise=true , address=77.90.31.15:4713 , piece_id=26S5ETN4OVNEL7IYYAIUBLP6UDS2QSSXCHD4H6YNHEQNY6FPOBOA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: read tcp 188.166.225.170:41506->77.90.31.15:4713: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811942720 (263h23m45.243022841s) [seg_pos={0 0}, upload=273834]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Aug 19 16:49:51.082762534 (263h23m45.513842665s) [upload=273834]: Done uploading segments... Aug 19 16:49:51.082792671 (263h23m45.513872803s) [upload=273834]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.998794483s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X....X........] duration: 6.248431776s FAIL (2025-08-19 16:55:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.248452476s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................................X....XX.........] duration: 6.396089548s FAIL (2025-08-19 16:55:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 19 16:49:12.801154313 (263h23m7.232234445s) [upload=273839]: Starting upload Aug 19 16:49:12.801215942 (263h23m7.232296073s) [upload=273839]: Done starting upload Aug 19 16:49:12.801330351 (263h23m7.232410482s) [upload=273839]: Uploading segments... Aug 19 16:49:12.801425372 (263h23m7.232505503s) [upload=273839]: Got next segment. Inline: false Aug 19 16:49:12.801455880 (263h23m7.232536012s) [upload=273839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:49:12.801461982 (263h23m7.232542113s) [upload=273839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:49:13.036708228 (263h23m7.467788359s) [seg_pos={0 0}, upload=273839]: Begin upload segment... Aug 19 16:49:13.036749197 (263h23m7.467829328s) [seg_pos={0 0}, upload=273839]: Done begin upload segment. Aug 19 16:49:13.036771971 (263h23m7.467852092s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.036780256 (263h23m7.467860437s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037135917 (263h23m7.468216048s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037326592 (263h23m7.468406723s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037403259 (263h23m7.46848339s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037667814 (263h23m7.468747945s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037761764 (263h23m7.468841896s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037943331 (263h23m7.469023462s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038070565 (263h23m7.469150696s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038168622 (263h23m7.469248753s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038300104 (263h23m7.469380235s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038396789 (263h23m7.46947692s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038622241 (263h23m7.469702362s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038679350 (263h23m7.469759481s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038854735 (263h23m7.469934856s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.039048045 (263h23m7.470128166s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:14.749064365 (263h23m9.180144496s) [upload=273839]: Next returned nil segment Aug 19 16:49:14.749117217 (263h23m9.180197348s) [upload=273839]: Waiting for error group managing segments... Aug 19 16:49:15.090317863 (263h23m9.521397994s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.132109007 (263h23m9.563189138s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.222090156 (263h23m9.653170287s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.328041780 (263h23m9.759121911s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.353998781 (263h23m9.785078912s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:36.709543777 (263h23m31.140623898s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:37.041062020 (263h23m31.472142141s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.183707791 (263h23m32.614787922s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.694075084 (263h23m33.125155215s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.716788163 (263h23m34.147868294s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.851890717 (263h23m34.282970848s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:40.293306092 (263h23m34.724386243s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:42.156929229 (263h23m36.58800936s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157075889 (263h23m36.58815601s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157126075 (263h23m36.588206206s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection Aug 19 16:49:42.157170951 (263h23m36.588251082s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: read tcp 188.166.225.170:34242->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:42.157215747 (263h23m36.588295878s) [seg_pos={0 0}, upload=273839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 19 16:49:42.385478364 (263h23m36.816558495s) [upload=273839]: Done uploading segments... Aug 19 16:49:42.385503091 (263h23m36.816583212s) [upload=273839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.396120346s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.................X.X...X...X..............X......X.XX.......] duration: 6.303712671s FAIL (2025-08-19 16:55:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Aug 19 16:48:51.452491912 (263h22m45.883572043s) [upload=273834]: Starting upload Aug 19 16:48:51.452531028 (263h22m45.883611159s) [upload=273834]: Done starting upload Aug 19 16:48:51.452781006 (263h22m45.883861137s) [upload=273834]: Uploading segments... Aug 19 16:48:51.452853734 (263h22m45.883933866s) [upload=273834]: Got next segment. Inline: false Aug 19 16:48:51.452892670 (263h22m45.883972871s) [upload=273834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:48:51.452899813 (263h22m45.883980004s) [upload=273834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:48:51.962807465 (263h22m46.393887596s) [seg_pos={0 0}, upload=273834]: Begin upload segment... Aug 19 16:48:51.962852621 (263h22m46.393932743s) [seg_pos={0 0}, upload=273834]: Done begin upload segment. Aug 19 16:48:51.962875075 (263h22m46.393955206s) [noise=true , address=104.36.82.23:4470 , piece_id=5PDSSXU4HWGHBN62JJQBKHDQ6YYJVBWZZYSCJGKOPTZAZZUCAB6Q, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.962917746 (263h22m46.393997877s) [noise=true , address=66.23.193.175:4556 , piece_id=FBS7KZJF3HULYZZJDNLNOKMR5HSCR224IY7ZU4FUKZC5YMIIOAJA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963271022 (263h22m46.394351153s) [noise=true , address=45.91.116.9:4286 , piece_id=WYJJULBB3QVG2F46C52OCJWAMHVORGI7MDQS3CSYLF7REGAZQ3OQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963372556 (263h22m46.394452687s) [noise=true , address=156.146.43.225:4452 , piece_id=3T6HWTUPTX6A7BZULB4RSKGUO6GNDIHQGUP5W3EESJBWZGN7LLEQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963580684 (263h22m46.394660815s) [noise=true , address=104.36.82.35:4606 , piece_id=R37F22AIPFPV45K2PRWAJNONXQEDBJQPZGQKQ3LSHAQIBPKRXLHA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963653934 (263h22m46.394734066s) [noise=true , address=104.36.82.21:4478 , piece_id=XJWKXZ45KPRM3QUYXVIYU75M24GOCWGG5E3K4V62225XF3TGSS4Q, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963841874 (263h22m46.394922005s) [noise=true , address=79.127.201.215:4343 , piece_id=JF5HESY3ZARRZWZJHGTIG2BEZWWMXPHNJF5ALABQBAOVGJCHKI5A, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963910455 (263h22m46.394990586s) [noise=true , address=79.127.219.51:4211 , piece_id=OUB6QKVWCRJATMQFLAQ7UK3LJ4DILSLRAGFDP5YXROEXHZCPWGZA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964097293 (263h22m46.395177424s) [noise=true , address=104.36.82.60:4341 , piece_id=JCAHPCHE5IPHX35ZCDQ27I3DZ3R347HFRQWNBCNBXU6P3O7PTONA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964163640 (263h22m46.395243771s) [noise=true , address=77.90.31.18:4828 , piece_id=NFXT5LHIVX4G6DXMB4UWI37XCXLNHBRLVLXSJWTYAYN2JKCHRZQA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964400383 (263h22m46.395480514s) [noise=true , address=104.36.82.51:4052 , piece_id=7HLVSHJCDKI3TNKFETBUAY25HVIOPKAMSYQ6Q5BJ2NCJTQOTETTA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964624241 (263h22m46.395704372s) [noise=true , address=77.90.31.10:4529 , piece_id=HK5DLHGNULVF2RX7NNKFYN5JIFJB3LDEYXLXO3UN4D5OO4DJOW4A, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964832199 (263h22m46.39591233s) [noise=true , address=45.91.116.11:4130 , piece_id=5IZILPAZ4UKTFE766TWKT672JHDPWIQJI3TMOTASODK7NYG5BKPQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965048673 (263h22m46.396128804s) [noise=true , address=104.36.82.11:4127 , piece_id=6ACOOH24TTEMA4OQP6OFNHIABWDW3NDUVD44FSRIFLPTSVPHXOOA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965259626 (263h22m46.396339747s) [noise=true , address=77.90.31.15:4713 , piece_id=26S5ETN4OVNEL7IYYAIUBLP6UDS2QSSXCHD4H6YNHEQNY6FPOBOA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965452415 (263h22m46.396532546s) [noise=true , address=45.91.116.14:4209 , piece_id=5KAIVWUEWAG5NW6QGOAJLIMUADHZ6R4YTNAKTTXAGBRUEYXDAE7A, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965641637 (263h22m46.396721768s) [noise=true , address=45.91.116.7:4020 , piece_id=5NNTOPQ5BC7T2TLQ7NWSO3E6R4NDPB3JPXPMPRGG6I7YJU6QS4CQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965869794 (263h22m46.396949925s) [noise=true , address=77.90.31.17:4793 , piece_id=K3GCAOBIKBJV6Q3KOFXL5SEUFYE46MKJU5IAYNDA25IDCICTOFCQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:53.162668249 (263h22m47.59374838s) [upload=273834]: Next returned nil segment Aug 19 16:48:53.162683478 (263h22m47.593763599s) [upload=273834]: Waiting for error group managing segments... Aug 19 16:48:53.488132069 (263h22m47.91921223s) [noise=true , address=66.23.193.175:4556 , piece_id=FBS7KZJF3HULYZZJDNLNOKMR5HSCR224IY7ZU4FUKZC5YMIIOAJA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.887114541 (263h22m48.318194672s) [noise=true , address=79.127.219.51:4211 , piece_id=OUB6QKVWCRJATMQFLAQ7UK3LJ4DILSLRAGFDP5YXROEXHZCPWGZA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.925197211 (263h22m48.356277342s) [noise=true , address=156.146.43.225:4452 , piece_id=3T6HWTUPTX6A7BZULB4RSKGUO6GNDIHQGUP5W3EESJBWZGN7LLEQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.961937701 (263h22m48.393017832s) [noise=true , address=104.36.82.11:4127 , piece_id=6ACOOH24TTEMA4OQP6OFNHIABWDW3NDUVD44FSRIFLPTSVPHXOOA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.992310049 (263h22m48.42339018s) [noise=true , address=79.127.201.215:4343 , piece_id=JF5HESY3ZARRZWZJHGTIG2BEZWWMXPHNJF5ALABQBAOVGJCHKI5A, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.045243820 (263h22m48.476323961s) [noise=true , address=104.36.82.35:4606 , piece_id=R37F22AIPFPV45K2PRWAJNONXQEDBJQPZGQKQ3LSHAQIBPKRXLHA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.059875193 (263h22m48.490955324s) [noise=true , address=104.36.82.21:4478 , piece_id=XJWKXZ45KPRM3QUYXVIYU75M24GOCWGG5E3K4V62225XF3TGSS4Q, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.095629989 (263h22m48.52671012s) [noise=true , address=104.36.82.23:4470 , piece_id=5PDSSXU4HWGHBN62JJQBKHDQ6YYJVBWZZYSCJGKOPTZAZZUCAB6Q, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.103250568 (263h22m48.534330699s) [noise=true , address=104.36.82.51:4052 , piece_id=7HLVSHJCDKI3TNKFETBUAY25HVIOPKAMSYQ6Q5BJ2NCJTQOTETTA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.159963800 (263h22m48.591043921s) [noise=true , address=104.36.82.60:4341 , piece_id=JCAHPCHE5IPHX35ZCDQ27I3DZ3R347HFRQWNBCNBXU6P3O7PTONA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:49:37.214809078 (263h23m31.64588921s) [noise=true , address=77.90.31.17:4793 , piece_id=K3GCAOBIKBJV6Q3KOFXL5SEUFYE46MKJU5IAYNDA25IDCICTOFCQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:49:50.811468753 (263h23m45.242548894s) [noise=true , address=77.90.31.10:4529 , piece_id=HK5DLHGNULVF2RX7NNKFYN5JIFJB3LDEYXLXO3UN4D5OO4DJOW4A, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: read tcp 188.166.225.170:57426->77.90.31.10:4529: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811576489 (263h23m45.24265662s) [noise=true , address=45.91.116.14:4209 , piece_id=5KAIVWUEWAG5NW6QGOAJLIMUADHZ6R4YTNAKTTXAGBRUEYXDAE7A, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:12LWbAxsrDcyV31SxJNiVNYs7ERKAf6RTuKpE596UrYJAnXMndR, address:45.91.116.14:4209): protocol: rpc: dial tcp 45.91.116.14:4209: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811688974 (263h23m45.242769105s) [noise=true , address=45.91.116.9:4286 , piece_id=WYJJULBB3QVG2F46C52OCJWAMHVORGI7MDQS3CSYLF7REGAZQ3OQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:19DfbHig9tzF4VRHtSWkExwUS2bdZ8GFgKnAD9wvMBG7qaesNa, address:45.91.116.9:4286): protocol: read tcp 188.166.225.170:40032->45.91.116.9:4286: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811727057 (263h23m45.242807188s) [noise=true , address=45.91.116.11:4130 , piece_id=5IZILPAZ4UKTFE766TWKT672JHDPWIQJI3TMOTASODK7NYG5BKPQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:126ThYkfUAYf2JR8f5rkwyumcvmRArxKbpLUjhJukoc3FADHj2T, address:45.91.116.11:4130): protocol: read tcp 188.166.225.170:46860->45.91.116.11:4130: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811808934 (263h23m45.242889085s) [noise=true , address=45.91.116.7:4020 , piece_id=5NNTOPQ5BC7T2TLQ7NWSO3E6R4NDPB3JPXPMPRGG6I7YJU6QS4CQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:1MT5Nk9DRHLLfHi5sJnzWEjuL3aV2HnbS9Xebdw8vtBMjQmFnV, address:45.91.116.7:4020): protocol: read tcp 188.166.225.170:48966->45.91.116.7:4020: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811874559 (263h23m45.24295469s) [noise=true , address=77.90.31.18:4828 , piece_id=NFXT5LHIVX4G6DXMB4UWI37XCXLNHBRLVLXSJWTYAYN2JKCHRZQA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:1jwTDXPShXfNdfMx3UXi22gQphBpFre1MhRsHyjq6xRz4wwTr9, address:77.90.31.18:4828): protocol: read tcp 188.166.225.170:47612->77.90.31.18:4828: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811914556 (263h23m45.242994687s) [noise=true , address=77.90.31.15:4713 , piece_id=26S5ETN4OVNEL7IYYAIUBLP6UDS2QSSXCHD4H6YNHEQNY6FPOBOA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: read tcp 188.166.225.170:41506->77.90.31.15:4713: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811942720 (263h23m45.243022841s) [seg_pos={0 0}, upload=273834]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Aug 19 16:49:51.082762534 (263h23m45.513842665s) [upload=273834]: Done uploading segments... Aug 19 16:49:51.082792671 (263h23m45.513872803s) [upload=273834]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 6.303739352s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 6.3069167s FAIL (2025-08-19 16:55:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 263h18m33.75518017s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324344006 (263h18m33.755424127s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324633220 (263h18m33.755713351s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324851286 (263h18m33.755931417s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325068942 (263h18m33.756149063s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325283613 (263h18m33.756363744s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325499857 (263h18m33.756579998s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325759794 (263h18m33.756839915s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326059428 (263h18m33.757139549s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326277565 (263h18m33.757357696s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326470835 (263h18m33.757550966s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326673753 (263h18m33.757753884s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326867664 (263h18m33.757947795s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:40.851885389 (263h18m35.28296552s) [upload=273764]: Next returned nil segment Aug 19 16:44:40.851903334 (263h18m35.282983465s) [upload=273764]: Waiting for error group managing segments... Aug 19 16:44:41.244668372 (263h18m35.675748503s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.277280365 (263h18m35.708360496s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.328561665 (263h18m35.759641786s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.331747139 (263h18m35.76282727s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.357720320 (263h18m35.788800481s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.378190805 (263h18m35.809270936s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.486480333 (263h18m35.917560464s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:45:38.230570787 (263h19m32.661650918s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: expected piece hash; context deadline exceeded; EOF Aug 19 16:45:38.230979869 (263h19m32.66206s) [noise=true , address=77.90.31.18:4840 , piece_id=JWOETSFSZ3HELLZNLEASYNGW5OR64TMQVOFYYJAJVTHWCBHVTSLQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: read tcp 188.166.225.170:39946->77.90.31.18:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231017812 (263h19m32.662097943s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 188.166.225.170:44164->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231063489 (263h19m32.66214362s) [noise=true , address=45.91.116.14:4203 , piece_id=WRLBES3FI5EA5AQLZXJYGDROP3KBG3E2F22R4YD6A5KKI44PLV2Q, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 188.166.225.170:51738->45.91.116.14:4203: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231090952 (263h19m32.662171083s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: read tcp 188.166.225.170:57158->77.90.31.15:4707: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231110719 (263h19m32.66219085s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:50784->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231140165 (263h19m32.662220296s) [noise=true , address=77.90.31.16:4749 , piece_id=XCPXVBYLQH345FPIW4KDNLBFUCDDEW2H4B25WIXGOO5FVMAIVKPQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: read tcp 188.166.225.170:51668->77.90.31.16:4749: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231144814 (263h19m32.662224945s) [noise=true , address=77.90.31.19:4865 , piece_id=B3HUFIJZKVZJH4OBBO2DD7S3I7HX4LO2IQ74XJPO4DTW4QER35NA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1bguBghQonV5sTHaWteyUEnpp5KNzBKbvBpbvrUNFDCveHN77X, address:77.90.31.19:4865): protocol: read tcp 188.166.225.170:39398->77.90.31.19:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231177167 (263h19m32.662257298s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: read tcp 188.166.225.170:58526->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231184571 (263h19m32.662264702s) [noise=true , address=45.91.116.12:4236 , piece_id=LCRQS2GOSHOSLZBGQTGB3Q3PA4VN7KXNW66TDZ5O424TPJWTQE3A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH, address:45.91.116.12:4236): protocol: read tcp 188.166.225.170:57166->45.91.116.12:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231189770 (263h19m32.662269901s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 188.166.225.170:54242->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231215850 (263h19m32.662295981s) [seg_pos={0 0}, upload=273764]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 19 16:45:38.459100173 (263h19m32.890180304s) [upload=273764]: Done uploading segments... Aug 19 16:45:38.459128316 (263h19m32.890208447s) [upload=273764]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 6.306950215s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X...X.........] duration: 6.711027363s FAIL (2025-08-19 16:55:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.711059734s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................................X....XX........] duration: 6.097934506s FAIL (2025-08-19 16:54:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 19 16:49:12.801154313 (263h23m7.232234445s) [upload=273839]: Starting upload Aug 19 16:49:12.801215942 (263h23m7.232296073s) [upload=273839]: Done starting upload Aug 19 16:49:12.801330351 (263h23m7.232410482s) [upload=273839]: Uploading segments... Aug 19 16:49:12.801425372 (263h23m7.232505503s) [upload=273839]: Got next segment. Inline: false Aug 19 16:49:12.801455880 (263h23m7.232536012s) [upload=273839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:49:12.801461982 (263h23m7.232542113s) [upload=273839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:49:13.036708228 (263h23m7.467788359s) [seg_pos={0 0}, upload=273839]: Begin upload segment... Aug 19 16:49:13.036749197 (263h23m7.467829328s) [seg_pos={0 0}, upload=273839]: Done begin upload segment. Aug 19 16:49:13.036771971 (263h23m7.467852092s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.036780256 (263h23m7.467860437s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037135917 (263h23m7.468216048s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037326592 (263h23m7.468406723s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037403259 (263h23m7.46848339s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037667814 (263h23m7.468747945s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037761764 (263h23m7.468841896s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037943331 (263h23m7.469023462s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038070565 (263h23m7.469150696s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038168622 (263h23m7.469248753s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038300104 (263h23m7.469380235s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038396789 (263h23m7.46947692s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038622241 (263h23m7.469702362s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038679350 (263h23m7.469759481s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038854735 (263h23m7.469934856s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.039048045 (263h23m7.470128166s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:14.749064365 (263h23m9.180144496s) [upload=273839]: Next returned nil segment Aug 19 16:49:14.749117217 (263h23m9.180197348s) [upload=273839]: Waiting for error group managing segments... Aug 19 16:49:15.090317863 (263h23m9.521397994s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.132109007 (263h23m9.563189138s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.222090156 (263h23m9.653170287s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.328041780 (263h23m9.759121911s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.353998781 (263h23m9.785078912s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:36.709543777 (263h23m31.140623898s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:37.041062020 (263h23m31.472142141s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.183707791 (263h23m32.614787922s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.694075084 (263h23m33.125155215s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.716788163 (263h23m34.147868294s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.851890717 (263h23m34.282970848s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:40.293306092 (263h23m34.724386243s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:42.156929229 (263h23m36.58800936s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157075889 (263h23m36.58815601s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157126075 (263h23m36.588206206s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection Aug 19 16:49:42.157170951 (263h23m36.588251082s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: read tcp 188.166.225.170:34242->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:42.157215747 (263h23m36.588295878s) [seg_pos={0 0}, upload=273839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 19 16:49:42.385478364 (263h23m36.816558495s) [upload=273839]: Done uploading segments... Aug 19 16:49:42.385503091 (263h23m36.816583212s) [upload=273839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.097962269s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..............X...............X.............................] duration: 2.679263881s FAIL (2025-08-19 16:54:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after be5a5f4d20475bc6e6586ca4cb41bd260785c25704d0f801265da5a966efbd34 != before fab796bcbc8cefce4dfe0fedb7e23ecb2129705e75b1b66c720b4c58f0a82c58 ======LOGS====== Aug 19 16:31:34.462024854 (263h5m28.893104995s) [upload=273541]: Starting upload Aug 19 16:31:34.462073225 (263h5m28.893153356s) [upload=273541]: Done starting upload Aug 19 16:31:34.462124303 (263h5m28.893204434s) [upload=273541]: Uploading segments... Aug 19 16:31:34.462159490 (263h5m28.893239621s) [upload=273541]: Got next segment. Inline: true Aug 19 16:31:34.462163108 (263h5m28.893243239s) [upload=273541]: Waiting for error group managing segments... Aug 19 16:31:34.462196912 (263h5m28.893277044s) [upload=273541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:31:34.462200970 (263h5m28.893281101s) [upload=273541]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:31:34.462203415 (263h5m28.893283546s) [upload=273541]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:31:34.760143563 (263h5m29.191223694s) [upload=273541]: Done uploading segments... Aug 19 16:31:34.760161496 (263h5m29.191241628s) [upload=273541]: Upload finished. err: <nil> Duration:* 2.6792943s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [..................X.X...X...X..............X......X.XX......] duration: 5.916841023s FAIL (2025-08-19 16:54:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Aug 19 16:48:51.452491912 (263h22m45.883572043s) [upload=273834]: Starting upload Aug 19 16:48:51.452531028 (263h22m45.883611159s) [upload=273834]: Done starting upload Aug 19 16:48:51.452781006 (263h22m45.883861137s) [upload=273834]: Uploading segments... Aug 19 16:48:51.452853734 (263h22m45.883933866s) [upload=273834]: Got next segment. Inline: false Aug 19 16:48:51.452892670 (263h22m45.883972871s) [upload=273834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:48:51.452899813 (263h22m45.883980004s) [upload=273834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:48:51.962807465 (263h22m46.393887596s) [seg_pos={0 0}, upload=273834]: Begin upload segment... Aug 19 16:48:51.962852621 (263h22m46.393932743s) [seg_pos={0 0}, upload=273834]: Done begin upload segment. Aug 19 16:48:51.962875075 (263h22m46.393955206s) [noise=true , address=104.36.82.23:4470 , piece_id=5PDSSXU4HWGHBN62JJQBKHDQ6YYJVBWZZYSCJGKOPTZAZZUCAB6Q, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.962917746 (263h22m46.393997877s) [noise=true , address=66.23.193.175:4556 , piece_id=FBS7KZJF3HULYZZJDNLNOKMR5HSCR224IY7ZU4FUKZC5YMIIOAJA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963271022 (263h22m46.394351153s) [noise=true , address=45.91.116.9:4286 , piece_id=WYJJULBB3QVG2F46C52OCJWAMHVORGI7MDQS3CSYLF7REGAZQ3OQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963372556 (263h22m46.394452687s) [noise=true , address=156.146.43.225:4452 , piece_id=3T6HWTUPTX6A7BZULB4RSKGUO6GNDIHQGUP5W3EESJBWZGN7LLEQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963580684 (263h22m46.394660815s) [noise=true , address=104.36.82.35:4606 , piece_id=R37F22AIPFPV45K2PRWAJNONXQEDBJQPZGQKQ3LSHAQIBPKRXLHA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963653934 (263h22m46.394734066s) [noise=true , address=104.36.82.21:4478 , piece_id=XJWKXZ45KPRM3QUYXVIYU75M24GOCWGG5E3K4V62225XF3TGSS4Q, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963841874 (263h22m46.394922005s) [noise=true , address=79.127.201.215:4343 , piece_id=JF5HESY3ZARRZWZJHGTIG2BEZWWMXPHNJF5ALABQBAOVGJCHKI5A, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963910455 (263h22m46.394990586s) [noise=true , address=79.127.219.51:4211 , piece_id=OUB6QKVWCRJATMQFLAQ7UK3LJ4DILSLRAGFDP5YXROEXHZCPWGZA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964097293 (263h22m46.395177424s) [noise=true , address=104.36.82.60:4341 , piece_id=JCAHPCHE5IPHX35ZCDQ27I3DZ3R347HFRQWNBCNBXU6P3O7PTONA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964163640 (263h22m46.395243771s) [noise=true , address=77.90.31.18:4828 , piece_id=NFXT5LHIVX4G6DXMB4UWI37XCXLNHBRLVLXSJWTYAYN2JKCHRZQA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964400383 (263h22m46.395480514s) [noise=true , address=104.36.82.51:4052 , piece_id=7HLVSHJCDKI3TNKFETBUAY25HVIOPKAMSYQ6Q5BJ2NCJTQOTETTA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964624241 (263h22m46.395704372s) [noise=true , address=77.90.31.10:4529 , piece_id=HK5DLHGNULVF2RX7NNKFYN5JIFJB3LDEYXLXO3UN4D5OO4DJOW4A, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964832199 (263h22m46.39591233s) [noise=true , address=45.91.116.11:4130 , piece_id=5IZILPAZ4UKTFE766TWKT672JHDPWIQJI3TMOTASODK7NYG5BKPQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965048673 (263h22m46.396128804s) [noise=true , address=104.36.82.11:4127 , piece_id=6ACOOH24TTEMA4OQP6OFNHIABWDW3NDUVD44FSRIFLPTSVPHXOOA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965259626 (263h22m46.396339747s) [noise=true , address=77.90.31.15:4713 , piece_id=26S5ETN4OVNEL7IYYAIUBLP6UDS2QSSXCHD4H6YNHEQNY6FPOBOA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965452415 (263h22m46.396532546s) [noise=true , address=45.91.116.14:4209 , piece_id=5KAIVWUEWAG5NW6QGOAJLIMUADHZ6R4YTNAKTTXAGBRUEYXDAE7A, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965641637 (263h22m46.396721768s) [noise=true , address=45.91.116.7:4020 , piece_id=5NNTOPQ5BC7T2TLQ7NWSO3E6R4NDPB3JPXPMPRGG6I7YJU6QS4CQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965869794 (263h22m46.396949925s) [noise=true , address=77.90.31.17:4793 , piece_id=K3GCAOBIKBJV6Q3KOFXL5SEUFYE46MKJU5IAYNDA25IDCICTOFCQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:53.162668249 (263h22m47.59374838s) [upload=273834]: Next returned nil segment Aug 19 16:48:53.162683478 (263h22m47.593763599s) [upload=273834]: Waiting for error group managing segments... Aug 19 16:48:53.488132069 (263h22m47.91921223s) [noise=true , address=66.23.193.175:4556 , piece_id=FBS7KZJF3HULYZZJDNLNOKMR5HSCR224IY7ZU4FUKZC5YMIIOAJA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.887114541 (263h22m48.318194672s) [noise=true , address=79.127.219.51:4211 , piece_id=OUB6QKVWCRJATMQFLAQ7UK3LJ4DILSLRAGFDP5YXROEXHZCPWGZA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.925197211 (263h22m48.356277342s) [noise=true , address=156.146.43.225:4452 , piece_id=3T6HWTUPTX6A7BZULB4RSKGUO6GNDIHQGUP5W3EESJBWZGN7LLEQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.961937701 (263h22m48.393017832s) [noise=true , address=104.36.82.11:4127 , piece_id=6ACOOH24TTEMA4OQP6OFNHIABWDW3NDUVD44FSRIFLPTSVPHXOOA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.992310049 (263h22m48.42339018s) [noise=true , address=79.127.201.215:4343 , piece_id=JF5HESY3ZARRZWZJHGTIG2BEZWWMXPHNJF5ALABQBAOVGJCHKI5A, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.045243820 (263h22m48.476323961s) [noise=true , address=104.36.82.35:4606 , piece_id=R37F22AIPFPV45K2PRWAJNONXQEDBJQPZGQKQ3LSHAQIBPKRXLHA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.059875193 (263h22m48.490955324s) [noise=true , address=104.36.82.21:4478 , piece_id=XJWKXZ45KPRM3QUYXVIYU75M24GOCWGG5E3K4V62225XF3TGSS4Q, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.095629989 (263h22m48.52671012s) [noise=true , address=104.36.82.23:4470 , piece_id=5PDSSXU4HWGHBN62JJQBKHDQ6YYJVBWZZYSCJGKOPTZAZZUCAB6Q, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.103250568 (263h22m48.534330699s) [noise=true , address=104.36.82.51:4052 , piece_id=7HLVSHJCDKI3TNKFETBUAY25HVIOPKAMSYQ6Q5BJ2NCJTQOTETTA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.159963800 (263h22m48.591043921s) [noise=true , address=104.36.82.60:4341 , piece_id=JCAHPCHE5IPHX35ZCDQ27I3DZ3R347HFRQWNBCNBXU6P3O7PTONA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:49:37.214809078 (263h23m31.64588921s) [noise=true , address=77.90.31.17:4793 , piece_id=K3GCAOBIKBJV6Q3KOFXL5SEUFYE46MKJU5IAYNDA25IDCICTOFCQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:49:50.811468753 (263h23m45.242548894s) [noise=true , address=77.90.31.10:4529 , piece_id=HK5DLHGNULVF2RX7NNKFYN5JIFJB3LDEYXLXO3UN4D5OO4DJOW4A, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: read tcp 188.166.225.170:57426->77.90.31.10:4529: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811576489 (263h23m45.24265662s) [noise=true , address=45.91.116.14:4209 , piece_id=5KAIVWUEWAG5NW6QGOAJLIMUADHZ6R4YTNAKTTXAGBRUEYXDAE7A, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:12LWbAxsrDcyV31SxJNiVNYs7ERKAf6RTuKpE596UrYJAnXMndR, address:45.91.116.14:4209): protocol: rpc: dial tcp 45.91.116.14:4209: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811688974 (263h23m45.242769105s) [noise=true , address=45.91.116.9:4286 , piece_id=WYJJULBB3QVG2F46C52OCJWAMHVORGI7MDQS3CSYLF7REGAZQ3OQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:19DfbHig9tzF4VRHtSWkExwUS2bdZ8GFgKnAD9wvMBG7qaesNa, address:45.91.116.9:4286): protocol: read tcp 188.166.225.170:40032->45.91.116.9:4286: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811727057 (263h23m45.242807188s) [noise=true , address=45.91.116.11:4130 , piece_id=5IZILPAZ4UKTFE766TWKT672JHDPWIQJI3TMOTASODK7NYG5BKPQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:126ThYkfUAYf2JR8f5rkwyumcvmRArxKbpLUjhJukoc3FADHj2T, address:45.91.116.11:4130): protocol: read tcp 188.166.225.170:46860->45.91.116.11:4130: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811808934 (263h23m45.242889085s) [noise=true , address=45.91.116.7:4020 , piece_id=5NNTOPQ5BC7T2TLQ7NWSO3E6R4NDPB3JPXPMPRGG6I7YJU6QS4CQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:1MT5Nk9DRHLLfHi5sJnzWEjuL3aV2HnbS9Xebdw8vtBMjQmFnV, address:45.91.116.7:4020): protocol: read tcp 188.166.225.170:48966->45.91.116.7:4020: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811874559 (263h23m45.24295469s) [noise=true , address=77.90.31.18:4828 , piece_id=NFXT5LHIVX4G6DXMB4UWI37XCXLNHBRLVLXSJWTYAYN2JKCHRZQA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:1jwTDXPShXfNdfMx3UXi22gQphBpFre1MhRsHyjq6xRz4wwTr9, address:77.90.31.18:4828): protocol: read tcp 188.166.225.170:47612->77.90.31.18:4828: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811914556 (263h23m45.242994687s) [noise=true , address=77.90.31.15:4713 , piece_id=26S5ETN4OVNEL7IYYAIUBLP6UDS2QSSXCHD4H6YNHEQNY6FPOBOA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: read tcp 188.166.225.170:41506->77.90.31.15:4713: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811942720 (263h23m45.243022841s) [seg_pos={0 0}, upload=273834]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Aug 19 16:49:51.082762534 (263h23m45.513842665s) [upload=273834]: Done uploading segments... Aug 19 16:49:51.082792671 (263h23m45.513872803s) [upload=273834]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.916867845s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X....X.......] duration: 6.013519508s FAIL (2025-08-19 16:54:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.013535148s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X...X........] duration: 6.509678825s FAIL (2025-08-19 16:54:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.509704314s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................................X....XX.......] duration: 6.041341924s FAIL (2025-08-19 16:54:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 19 16:49:12.801154313 (263h23m7.232234445s) [upload=273839]: Starting upload Aug 19 16:49:12.801215942 (263h23m7.232296073s) [upload=273839]: Done starting upload Aug 19 16:49:12.801330351 (263h23m7.232410482s) [upload=273839]: Uploading segments... Aug 19 16:49:12.801425372 (263h23m7.232505503s) [upload=273839]: Got next segment. Inline: false Aug 19 16:49:12.801455880 (263h23m7.232536012s) [upload=273839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:49:12.801461982 (263h23m7.232542113s) [upload=273839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:49:13.036708228 (263h23m7.467788359s) [seg_pos={0 0}, upload=273839]: Begin upload segment... Aug 19 16:49:13.036749197 (263h23m7.467829328s) [seg_pos={0 0}, upload=273839]: Done begin upload segment. Aug 19 16:49:13.036771971 (263h23m7.467852092s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.036780256 (263h23m7.467860437s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037135917 (263h23m7.468216048s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037326592 (263h23m7.468406723s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037403259 (263h23m7.46848339s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037667814 (263h23m7.468747945s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037761764 (263h23m7.468841896s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037943331 (263h23m7.469023462s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038070565 (263h23m7.469150696s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038168622 (263h23m7.469248753s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038300104 (263h23m7.469380235s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038396789 (263h23m7.46947692s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038622241 (263h23m7.469702362s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038679350 (263h23m7.469759481s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038854735 (263h23m7.469934856s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.039048045 (263h23m7.470128166s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:14.749064365 (263h23m9.180144496s) [upload=273839]: Next returned nil segment Aug 19 16:49:14.749117217 (263h23m9.180197348s) [upload=273839]: Waiting for error group managing segments... Aug 19 16:49:15.090317863 (263h23m9.521397994s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.132109007 (263h23m9.563189138s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.222090156 (263h23m9.653170287s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.328041780 (263h23m9.759121911s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.353998781 (263h23m9.785078912s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:36.709543777 (263h23m31.140623898s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:37.041062020 (263h23m31.472142141s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.183707791 (263h23m32.614787922s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.694075084 (263h23m33.125155215s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.716788163 (263h23m34.147868294s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.851890717 (263h23m34.282970848s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:40.293306092 (263h23m34.724386243s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:42.156929229 (263h23m36.58800936s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157075889 (263h23m36.58815601s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157126075 (263h23m36.588206206s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection Aug 19 16:49:42.157170951 (263h23m36.588251082s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: read tcp 188.166.225.170:34242->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:42.157215747 (263h23m36.588295878s) [seg_pos={0 0}, upload=273839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 19 16:49:42.385478364 (263h23m36.816558495s) [upload=273839]: Done uploading segments... Aug 19 16:49:42.385503091 (263h23m36.816583212s) [upload=273839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.041365068s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.756192269s FAIL (2025-08-19 16:54:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 263h18m33.75518017s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324344006 (263h18m33.755424127s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324633220 (263h18m33.755713351s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324851286 (263h18m33.755931417s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325068942 (263h18m33.756149063s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325283613 (263h18m33.756363744s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325499857 (263h18m33.756579998s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325759794 (263h18m33.756839915s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326059428 (263h18m33.757139549s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326277565 (263h18m33.757357696s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326470835 (263h18m33.757550966s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326673753 (263h18m33.757753884s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326867664 (263h18m33.757947795s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:40.851885389 (263h18m35.28296552s) [upload=273764]: Next returned nil segment Aug 19 16:44:40.851903334 (263h18m35.282983465s) [upload=273764]: Waiting for error group managing segments... Aug 19 16:44:41.244668372 (263h18m35.675748503s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.277280365 (263h18m35.708360496s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.328561665 (263h18m35.759641786s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.331747139 (263h18m35.76282727s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.357720320 (263h18m35.788800481s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.378190805 (263h18m35.809270936s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.486480333 (263h18m35.917560464s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:45:38.230570787 (263h19m32.661650918s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: expected piece hash; context deadline exceeded; EOF Aug 19 16:45:38.230979869 (263h19m32.66206s) [noise=true , address=77.90.31.18:4840 , piece_id=JWOETSFSZ3HELLZNLEASYNGW5OR64TMQVOFYYJAJVTHWCBHVTSLQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: read tcp 188.166.225.170:39946->77.90.31.18:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231017812 (263h19m32.662097943s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 188.166.225.170:44164->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231063489 (263h19m32.66214362s) [noise=true , address=45.91.116.14:4203 , piece_id=WRLBES3FI5EA5AQLZXJYGDROP3KBG3E2F22R4YD6A5KKI44PLV2Q, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 188.166.225.170:51738->45.91.116.14:4203: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231090952 (263h19m32.662171083s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: read tcp 188.166.225.170:57158->77.90.31.15:4707: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231110719 (263h19m32.66219085s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:50784->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231140165 (263h19m32.662220296s) [noise=true , address=77.90.31.16:4749 , piece_id=XCPXVBYLQH345FPIW4KDNLBFUCDDEW2H4B25WIXGOO5FVMAIVKPQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: read tcp 188.166.225.170:51668->77.90.31.16:4749: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231144814 (263h19m32.662224945s) [noise=true , address=77.90.31.19:4865 , piece_id=B3HUFIJZKVZJH4OBBO2DD7S3I7HX4LO2IQ74XJPO4DTW4QER35NA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1bguBghQonV5sTHaWteyUEnpp5KNzBKbvBpbvrUNFDCveHN77X, address:77.90.31.19:4865): protocol: read tcp 188.166.225.170:39398->77.90.31.19:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231177167 (263h19m32.662257298s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: read tcp 188.166.225.170:58526->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231184571 (263h19m32.662264702s) [noise=true , address=45.91.116.12:4236 , piece_id=LCRQS2GOSHOSLZBGQTGB3Q3PA4VN7KXNW66TDZ5O424TPJWTQE3A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH, address:45.91.116.12:4236): protocol: read tcp 188.166.225.170:57166->45.91.116.12:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231189770 (263h19m32.662269901s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 188.166.225.170:54242->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231215850 (263h19m32.662295981s) [seg_pos={0 0}, upload=273764]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 19 16:45:38.459100173 (263h19m32.890180304s) [upload=273764]: Done uploading segments... Aug 19 16:45:38.459128316 (263h19m32.890208447s) [upload=273764]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 6.756209983s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X....X......] duration: 5.806419815s FAIL (2025-08-19 16:54:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.806443469s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................................X....XX......] duration: 6.522543853s FAIL (2025-08-19 16:53:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 19 16:49:12.801154313 (263h23m7.232234445s) [upload=273839]: Starting upload Aug 19 16:49:12.801215942 (263h23m7.232296073s) [upload=273839]: Done starting upload Aug 19 16:49:12.801330351 (263h23m7.232410482s) [upload=273839]: Uploading segments... Aug 19 16:49:12.801425372 (263h23m7.232505503s) [upload=273839]: Got next segment. Inline: false Aug 19 16:49:12.801455880 (263h23m7.232536012s) [upload=273839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:49:12.801461982 (263h23m7.232542113s) [upload=273839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:49:13.036708228 (263h23m7.467788359s) [seg_pos={0 0}, upload=273839]: Begin upload segment... Aug 19 16:49:13.036749197 (263h23m7.467829328s) [seg_pos={0 0}, upload=273839]: Done begin upload segment. Aug 19 16:49:13.036771971 (263h23m7.467852092s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.036780256 (263h23m7.467860437s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037135917 (263h23m7.468216048s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037326592 (263h23m7.468406723s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037403259 (263h23m7.46848339s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037667814 (263h23m7.468747945s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037761764 (263h23m7.468841896s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037943331 (263h23m7.469023462s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038070565 (263h23m7.469150696s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038168622 (263h23m7.469248753s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038300104 (263h23m7.469380235s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038396789 (263h23m7.46947692s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038622241 (263h23m7.469702362s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038679350 (263h23m7.469759481s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038854735 (263h23m7.469934856s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.039048045 (263h23m7.470128166s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:14.749064365 (263h23m9.180144496s) [upload=273839]: Next returned nil segment Aug 19 16:49:14.749117217 (263h23m9.180197348s) [upload=273839]: Waiting for error group managing segments... Aug 19 16:49:15.090317863 (263h23m9.521397994s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.132109007 (263h23m9.563189138s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.222090156 (263h23m9.653170287s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.328041780 (263h23m9.759121911s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.353998781 (263h23m9.785078912s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:36.709543777 (263h23m31.140623898s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:37.041062020 (263h23m31.472142141s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.183707791 (263h23m32.614787922s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.694075084 (263h23m33.125155215s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.716788163 (263h23m34.147868294s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.851890717 (263h23m34.282970848s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:40.293306092 (263h23m34.724386243s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:42.156929229 (263h23m36.58800936s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157075889 (263h23m36.58815601s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157126075 (263h23m36.588206206s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection Aug 19 16:49:42.157170951 (263h23m36.588251082s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: read tcp 188.166.225.170:34242->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:42.157215747 (263h23m36.588295878s) [seg_pos={0 0}, upload=273839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 19 16:49:42.385478364 (263h23m36.816558495s) [upload=273839]: Done uploading segments... Aug 19 16:49:42.385503091 (263h23m36.816583212s) [upload=273839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.522569613s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X....X.....] duration: 6.16928382s FAIL (2025-08-19 16:53:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.169303188s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...............X...............X............................] duration: 1.832534809s FAIL (2025-08-19 16:53:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after be5a5f4d20475bc6e6586ca4cb41bd260785c25704d0f801265da5a966efbd34 != before fab796bcbc8cefce4dfe0fedb7e23ecb2129705e75b1b66c720b4c58f0a82c58 ======LOGS====== Aug 19 16:31:34.462024854 (263h5m28.893104995s) [upload=273541]: Starting upload Aug 19 16:31:34.462073225 (263h5m28.893153356s) [upload=273541]: Done starting upload Aug 19 16:31:34.462124303 (263h5m28.893204434s) [upload=273541]: Uploading segments... Aug 19 16:31:34.462159490 (263h5m28.893239621s) [upload=273541]: Got next segment. Inline: true Aug 19 16:31:34.462163108 (263h5m28.893243239s) [upload=273541]: Waiting for error group managing segments... Aug 19 16:31:34.462196912 (263h5m28.893277044s) [upload=273541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:31:34.462200970 (263h5m28.893281101s) [upload=273541]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:31:34.462203415 (263h5m28.893283546s) [upload=273541]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:31:34.760143563 (263h5m29.191223694s) [upload=273541]: Done uploading segments... Aug 19 16:31:34.760161496 (263h5m29.191241628s) [upload=273541]: Upload finished. err: <nil> Duration:* 1.832563484s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................................................X...X.......] duration: 6.44922004s FAIL (2025-08-19 16:53:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.449241952s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [...................X.X...X...X..............X......X.XX.....] duration: 5.827927336s FAIL (2025-08-19 16:53:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Aug 19 16:48:51.452491912 (263h22m45.883572043s) [upload=273834]: Starting upload Aug 19 16:48:51.452531028 (263h22m45.883611159s) [upload=273834]: Done starting upload Aug 19 16:48:51.452781006 (263h22m45.883861137s) [upload=273834]: Uploading segments... Aug 19 16:48:51.452853734 (263h22m45.883933866s) [upload=273834]: Got next segment. Inline: false Aug 19 16:48:51.452892670 (263h22m45.883972871s) [upload=273834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:48:51.452899813 (263h22m45.883980004s) [upload=273834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:48:51.962807465 (263h22m46.393887596s) [seg_pos={0 0}, upload=273834]: Begin upload segment... Aug 19 16:48:51.962852621 (263h22m46.393932743s) [seg_pos={0 0}, upload=273834]: Done begin upload segment. Aug 19 16:48:51.962875075 (263h22m46.393955206s) [noise=true , address=104.36.82.23:4470 , piece_id=5PDSSXU4HWGHBN62JJQBKHDQ6YYJVBWZZYSCJGKOPTZAZZUCAB6Q, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.962917746 (263h22m46.393997877s) [noise=true , address=66.23.193.175:4556 , piece_id=FBS7KZJF3HULYZZJDNLNOKMR5HSCR224IY7ZU4FUKZC5YMIIOAJA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963271022 (263h22m46.394351153s) [noise=true , address=45.91.116.9:4286 , piece_id=WYJJULBB3QVG2F46C52OCJWAMHVORGI7MDQS3CSYLF7REGAZQ3OQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963372556 (263h22m46.394452687s) [noise=true , address=156.146.43.225:4452 , piece_id=3T6HWTUPTX6A7BZULB4RSKGUO6GNDIHQGUP5W3EESJBWZGN7LLEQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963580684 (263h22m46.394660815s) [noise=true , address=104.36.82.35:4606 , piece_id=R37F22AIPFPV45K2PRWAJNONXQEDBJQPZGQKQ3LSHAQIBPKRXLHA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963653934 (263h22m46.394734066s) [noise=true , address=104.36.82.21:4478 , piece_id=XJWKXZ45KPRM3QUYXVIYU75M24GOCWGG5E3K4V62225XF3TGSS4Q, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963841874 (263h22m46.394922005s) [noise=true , address=79.127.201.215:4343 , piece_id=JF5HESY3ZARRZWZJHGTIG2BEZWWMXPHNJF5ALABQBAOVGJCHKI5A, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963910455 (263h22m46.394990586s) [noise=true , address=79.127.219.51:4211 , piece_id=OUB6QKVWCRJATMQFLAQ7UK3LJ4DILSLRAGFDP5YXROEXHZCPWGZA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964097293 (263h22m46.395177424s) [noise=true , address=104.36.82.60:4341 , piece_id=JCAHPCHE5IPHX35ZCDQ27I3DZ3R347HFRQWNBCNBXU6P3O7PTONA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964163640 (263h22m46.395243771s) [noise=true , address=77.90.31.18:4828 , piece_id=NFXT5LHIVX4G6DXMB4UWI37XCXLNHBRLVLXSJWTYAYN2JKCHRZQA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964400383 (263h22m46.395480514s) [noise=true , address=104.36.82.51:4052 , piece_id=7HLVSHJCDKI3TNKFETBUAY25HVIOPKAMSYQ6Q5BJ2NCJTQOTETTA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964624241 (263h22m46.395704372s) [noise=true , address=77.90.31.10:4529 , piece_id=HK5DLHGNULVF2RX7NNKFYN5JIFJB3LDEYXLXO3UN4D5OO4DJOW4A, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964832199 (263h22m46.39591233s) [noise=true , address=45.91.116.11:4130 , piece_id=5IZILPAZ4UKTFE766TWKT672JHDPWIQJI3TMOTASODK7NYG5BKPQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965048673 (263h22m46.396128804s) [noise=true , address=104.36.82.11:4127 , piece_id=6ACOOH24TTEMA4OQP6OFNHIABWDW3NDUVD44FSRIFLPTSVPHXOOA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965259626 (263h22m46.396339747s) [noise=true , address=77.90.31.15:4713 , piece_id=26S5ETN4OVNEL7IYYAIUBLP6UDS2QSSXCHD4H6YNHEQNY6FPOBOA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965452415 (263h22m46.396532546s) [noise=true , address=45.91.116.14:4209 , piece_id=5KAIVWUEWAG5NW6QGOAJLIMUADHZ6R4YTNAKTTXAGBRUEYXDAE7A, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965641637 (263h22m46.396721768s) [noise=true , address=45.91.116.7:4020 , piece_id=5NNTOPQ5BC7T2TLQ7NWSO3E6R4NDPB3JPXPMPRGG6I7YJU6QS4CQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965869794 (263h22m46.396949925s) [noise=true , address=77.90.31.17:4793 , piece_id=K3GCAOBIKBJV6Q3KOFXL5SEUFYE46MKJU5IAYNDA25IDCICTOFCQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:53.162668249 (263h22m47.59374838s) [upload=273834]: Next returned nil segment Aug 19 16:48:53.162683478 (263h22m47.593763599s) [upload=273834]: Waiting for error group managing segments... Aug 19 16:48:53.488132069 (263h22m47.91921223s) [noise=true , address=66.23.193.175:4556 , piece_id=FBS7KZJF3HULYZZJDNLNOKMR5HSCR224IY7ZU4FUKZC5YMIIOAJA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.887114541 (263h22m48.318194672s) [noise=true , address=79.127.219.51:4211 , piece_id=OUB6QKVWCRJATMQFLAQ7UK3LJ4DILSLRAGFDP5YXROEXHZCPWGZA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.925197211 (263h22m48.356277342s) [noise=true , address=156.146.43.225:4452 , piece_id=3T6HWTUPTX6A7BZULB4RSKGUO6GNDIHQGUP5W3EESJBWZGN7LLEQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.961937701 (263h22m48.393017832s) [noise=true , address=104.36.82.11:4127 , piece_id=6ACOOH24TTEMA4OQP6OFNHIABWDW3NDUVD44FSRIFLPTSVPHXOOA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.992310049 (263h22m48.42339018s) [noise=true , address=79.127.201.215:4343 , piece_id=JF5HESY3ZARRZWZJHGTIG2BEZWWMXPHNJF5ALABQBAOVGJCHKI5A, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.045243820 (263h22m48.476323961s) [noise=true , address=104.36.82.35:4606 , piece_id=R37F22AIPFPV45K2PRWAJNONXQEDBJQPZGQKQ3LSHAQIBPKRXLHA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.059875193 (263h22m48.490955324s) [noise=true , address=104.36.82.21:4478 , piece_id=XJWKXZ45KPRM3QUYXVIYU75M24GOCWGG5E3K4V62225XF3TGSS4Q, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.095629989 (263h22m48.52671012s) [noise=true , address=104.36.82.23:4470 , piece_id=5PDSSXU4HWGHBN62JJQBKHDQ6YYJVBWZZYSCJGKOPTZAZZUCAB6Q, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.103250568 (263h22m48.534330699s) [noise=true , address=104.36.82.51:4052 , piece_id=7HLVSHJCDKI3TNKFETBUAY25HVIOPKAMSYQ6Q5BJ2NCJTQOTETTA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.159963800 (263h22m48.591043921s) [noise=true , address=104.36.82.60:4341 , piece_id=JCAHPCHE5IPHX35ZCDQ27I3DZ3R347HFRQWNBCNBXU6P3O7PTONA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:49:37.214809078 (263h23m31.64588921s) [noise=true , address=77.90.31.17:4793 , piece_id=K3GCAOBIKBJV6Q3KOFXL5SEUFYE46MKJU5IAYNDA25IDCICTOFCQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:49:50.811468753 (263h23m45.242548894s) [noise=true , address=77.90.31.10:4529 , piece_id=HK5DLHGNULVF2RX7NNKFYN5JIFJB3LDEYXLXO3UN4D5OO4DJOW4A, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: read tcp 188.166.225.170:57426->77.90.31.10:4529: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811576489 (263h23m45.24265662s) [noise=true , address=45.91.116.14:4209 , piece_id=5KAIVWUEWAG5NW6QGOAJLIMUADHZ6R4YTNAKTTXAGBRUEYXDAE7A, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:12LWbAxsrDcyV31SxJNiVNYs7ERKAf6RTuKpE596UrYJAnXMndR, address:45.91.116.14:4209): protocol: rpc: dial tcp 45.91.116.14:4209: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811688974 (263h23m45.242769105s) [noise=true , address=45.91.116.9:4286 , piece_id=WYJJULBB3QVG2F46C52OCJWAMHVORGI7MDQS3CSYLF7REGAZQ3OQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:19DfbHig9tzF4VRHtSWkExwUS2bdZ8GFgKnAD9wvMBG7qaesNa, address:45.91.116.9:4286): protocol: read tcp 188.166.225.170:40032->45.91.116.9:4286: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811727057 (263h23m45.242807188s) [noise=true , address=45.91.116.11:4130 , piece_id=5IZILPAZ4UKTFE766TWKT672JHDPWIQJI3TMOTASODK7NYG5BKPQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:126ThYkfUAYf2JR8f5rkwyumcvmRArxKbpLUjhJukoc3FADHj2T, address:45.91.116.11:4130): protocol: read tcp 188.166.225.170:46860->45.91.116.11:4130: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811808934 (263h23m45.242889085s) [noise=true , address=45.91.116.7:4020 , piece_id=5NNTOPQ5BC7T2TLQ7NWSO3E6R4NDPB3JPXPMPRGG6I7YJU6QS4CQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:1MT5Nk9DRHLLfHi5sJnzWEjuL3aV2HnbS9Xebdw8vtBMjQmFnV, address:45.91.116.7:4020): protocol: read tcp 188.166.225.170:48966->45.91.116.7:4020: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811874559 (263h23m45.24295469s) [noise=true , address=77.90.31.18:4828 , piece_id=NFXT5LHIVX4G6DXMB4UWI37XCXLNHBRLVLXSJWTYAYN2JKCHRZQA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:1jwTDXPShXfNdfMx3UXi22gQphBpFre1MhRsHyjq6xRz4wwTr9, address:77.90.31.18:4828): protocol: read tcp 188.166.225.170:47612->77.90.31.18:4828: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811914556 (263h23m45.242994687s) [noise=true , address=77.90.31.15:4713 , piece_id=26S5ETN4OVNEL7IYYAIUBLP6UDS2QSSXCHD4H6YNHEQNY6FPOBOA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: read tcp 188.166.225.170:41506->77.90.31.15:4713: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811942720 (263h23m45.243022841s) [seg_pos={0 0}, upload=273834]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Aug 19 16:49:51.082762534 (263h23m45.513842665s) [upload=273834]: Done uploading segments... Aug 19 16:49:51.082792671 (263h23m45.513872803s) [upload=273834]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.827956292s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X...X......] duration: 7.376921987s FAIL (2025-08-19 16:53:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.37696001s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.226109395s FAIL (2025-08-19 16:53:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 263h18m33.75518017s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324344006 (263h18m33.755424127s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324633220 (263h18m33.755713351s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324851286 (263h18m33.755931417s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325068942 (263h18m33.756149063s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325283613 (263h18m33.756363744s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325499857 (263h18m33.756579998s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325759794 (263h18m33.756839915s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326059428 (263h18m33.757139549s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326277565 (263h18m33.757357696s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326470835 (263h18m33.757550966s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326673753 (263h18m33.757753884s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326867664 (263h18m33.757947795s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:40.851885389 (263h18m35.28296552s) [upload=273764]: Next returned nil segment Aug 19 16:44:40.851903334 (263h18m35.282983465s) [upload=273764]: Waiting for error group managing segments... Aug 19 16:44:41.244668372 (263h18m35.675748503s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.277280365 (263h18m35.708360496s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.328561665 (263h18m35.759641786s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.331747139 (263h18m35.76282727s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.357720320 (263h18m35.788800481s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.378190805 (263h18m35.809270936s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.486480333 (263h18m35.917560464s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:45:38.230570787 (263h19m32.661650918s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: expected piece hash; context deadline exceeded; EOF Aug 19 16:45:38.230979869 (263h19m32.66206s) [noise=true , address=77.90.31.18:4840 , piece_id=JWOETSFSZ3HELLZNLEASYNGW5OR64TMQVOFYYJAJVTHWCBHVTSLQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: read tcp 188.166.225.170:39946->77.90.31.18:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231017812 (263h19m32.662097943s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 188.166.225.170:44164->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231063489 (263h19m32.66214362s) [noise=true , address=45.91.116.14:4203 , piece_id=WRLBES3FI5EA5AQLZXJYGDROP3KBG3E2F22R4YD6A5KKI44PLV2Q, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 188.166.225.170:51738->45.91.116.14:4203: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231090952 (263h19m32.662171083s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: read tcp 188.166.225.170:57158->77.90.31.15:4707: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231110719 (263h19m32.66219085s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:50784->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231140165 (263h19m32.662220296s) [noise=true , address=77.90.31.16:4749 , piece_id=XCPXVBYLQH345FPIW4KDNLBFUCDDEW2H4B25WIXGOO5FVMAIVKPQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: read tcp 188.166.225.170:51668->77.90.31.16:4749: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231144814 (263h19m32.662224945s) [noise=true , address=77.90.31.19:4865 , piece_id=B3HUFIJZKVZJH4OBBO2DD7S3I7HX4LO2IQ74XJPO4DTW4QER35NA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1bguBghQonV5sTHaWteyUEnpp5KNzBKbvBpbvrUNFDCveHN77X, address:77.90.31.19:4865): protocol: read tcp 188.166.225.170:39398->77.90.31.19:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231177167 (263h19m32.662257298s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: read tcp 188.166.225.170:58526->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231184571 (263h19m32.662264702s) [noise=true , address=45.91.116.12:4236 , piece_id=LCRQS2GOSHOSLZBGQTGB3Q3PA4VN7KXNW66TDZ5O424TPJWTQE3A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH, address:45.91.116.12:4236): protocol: read tcp 188.166.225.170:57166->45.91.116.12:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231189770 (263h19m32.662269901s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 188.166.225.170:54242->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231215850 (263h19m32.662295981s) [seg_pos={0 0}, upload=273764]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 19 16:45:38.459100173 (263h19m32.890180304s) [upload=273764]: Done uploading segments... Aug 19 16:45:38.459128316 (263h19m32.890208447s) [upload=273764]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 6.226132779s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [................X...............X...........................] duration: 2.447612265s FAIL (2025-08-19 16:52:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after be5a5f4d20475bc6e6586ca4cb41bd260785c25704d0f801265da5a966efbd34 != before fab796bcbc8cefce4dfe0fedb7e23ecb2129705e75b1b66c720b4c58f0a82c58 ======LOGS====== Aug 19 16:31:34.462024854 (263h5m28.893104995s) [upload=273541]: Starting upload Aug 19 16:31:34.462073225 (263h5m28.893153356s) [upload=273541]: Done starting upload Aug 19 16:31:34.462124303 (263h5m28.893204434s) [upload=273541]: Uploading segments... Aug 19 16:31:34.462159490 (263h5m28.893239621s) [upload=273541]: Got next segment. Inline: true Aug 19 16:31:34.462163108 (263h5m28.893243239s) [upload=273541]: Waiting for error group managing segments... Aug 19 16:31:34.462196912 (263h5m28.893277044s) [upload=273541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:31:34.462200970 (263h5m28.893281101s) [upload=273541]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:31:34.462203415 (263h5m28.893283546s) [upload=273541]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:31:34.760143563 (263h5m29.191223694s) [upload=273541]: Done uploading segments... Aug 19 16:31:34.760161496 (263h5m29.191241628s) [upload=273541]: Upload finished. err: <nil> Duration:* 2.447641631s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................................X....XX.....] duration: 6.486344054s FAIL (2025-08-19 16:52:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 19 16:49:12.801154313 (263h23m7.232234445s) [upload=273839]: Starting upload Aug 19 16:49:12.801215942 (263h23m7.232296073s) [upload=273839]: Done starting upload Aug 19 16:49:12.801330351 (263h23m7.232410482s) [upload=273839]: Uploading segments... Aug 19 16:49:12.801425372 (263h23m7.232505503s) [upload=273839]: Got next segment. Inline: false Aug 19 16:49:12.801455880 (263h23m7.232536012s) [upload=273839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:49:12.801461982 (263h23m7.232542113s) [upload=273839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:49:13.036708228 (263h23m7.467788359s) [seg_pos={0 0}, upload=273839]: Begin upload segment... Aug 19 16:49:13.036749197 (263h23m7.467829328s) [seg_pos={0 0}, upload=273839]: Done begin upload segment. Aug 19 16:49:13.036771971 (263h23m7.467852092s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.036780256 (263h23m7.467860437s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037135917 (263h23m7.468216048s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037326592 (263h23m7.468406723s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037403259 (263h23m7.46848339s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037667814 (263h23m7.468747945s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037761764 (263h23m7.468841896s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037943331 (263h23m7.469023462s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038070565 (263h23m7.469150696s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038168622 (263h23m7.469248753s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038300104 (263h23m7.469380235s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038396789 (263h23m7.46947692s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038622241 (263h23m7.469702362s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038679350 (263h23m7.469759481s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038854735 (263h23m7.469934856s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.039048045 (263h23m7.470128166s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:14.749064365 (263h23m9.180144496s) [upload=273839]: Next returned nil segment Aug 19 16:49:14.749117217 (263h23m9.180197348s) [upload=273839]: Waiting for error group managing segments... Aug 19 16:49:15.090317863 (263h23m9.521397994s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.132109007 (263h23m9.563189138s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.222090156 (263h23m9.653170287s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.328041780 (263h23m9.759121911s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.353998781 (263h23m9.785078912s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:36.709543777 (263h23m31.140623898s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:37.041062020 (263h23m31.472142141s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.183707791 (263h23m32.614787922s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.694075084 (263h23m33.125155215s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.716788163 (263h23m34.147868294s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.851890717 (263h23m34.282970848s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:40.293306092 (263h23m34.724386243s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:42.156929229 (263h23m36.58800936s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157075889 (263h23m36.58815601s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157126075 (263h23m36.588206206s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection Aug 19 16:49:42.157170951 (263h23m36.588251082s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: read tcp 188.166.225.170:34242->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:42.157215747 (263h23m36.588295878s) [seg_pos={0 0}, upload=273839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 19 16:49:42.385478364 (263h23m36.816558495s) [upload=273839]: Done uploading segments... Aug 19 16:49:42.385503091 (263h23m36.816583212s) [upload=273839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.486365465s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X....X....] duration: 5.922994888s FAIL (2025-08-19 16:52:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.923050274s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................................X....XX....] duration: 6.94801448s FAIL (2025-08-19 16:52:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 19 16:49:12.801154313 (263h23m7.232234445s) [upload=273839]: Starting upload Aug 19 16:49:12.801215942 (263h23m7.232296073s) [upload=273839]: Done starting upload Aug 19 16:49:12.801330351 (263h23m7.232410482s) [upload=273839]: Uploading segments... Aug 19 16:49:12.801425372 (263h23m7.232505503s) [upload=273839]: Got next segment. Inline: false Aug 19 16:49:12.801455880 (263h23m7.232536012s) [upload=273839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:49:12.801461982 (263h23m7.232542113s) [upload=273839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:49:13.036708228 (263h23m7.467788359s) [seg_pos={0 0}, upload=273839]: Begin upload segment... Aug 19 16:49:13.036749197 (263h23m7.467829328s) [seg_pos={0 0}, upload=273839]: Done begin upload segment. Aug 19 16:49:13.036771971 (263h23m7.467852092s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.036780256 (263h23m7.467860437s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037135917 (263h23m7.468216048s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037326592 (263h23m7.468406723s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037403259 (263h23m7.46848339s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037667814 (263h23m7.468747945s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037761764 (263h23m7.468841896s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037943331 (263h23m7.469023462s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038070565 (263h23m7.469150696s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038168622 (263h23m7.469248753s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038300104 (263h23m7.469380235s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038396789 (263h23m7.46947692s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038622241 (263h23m7.469702362s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038679350 (263h23m7.469759481s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038854735 (263h23m7.469934856s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.039048045 (263h23m7.470128166s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:14.749064365 (263h23m9.180144496s) [upload=273839]: Next returned nil segment Aug 19 16:49:14.749117217 (263h23m9.180197348s) [upload=273839]: Waiting for error group managing segments... Aug 19 16:49:15.090317863 (263h23m9.521397994s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.132109007 (263h23m9.563189138s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.222090156 (263h23m9.653170287s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.328041780 (263h23m9.759121911s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.353998781 (263h23m9.785078912s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:36.709543777 (263h23m31.140623898s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:37.041062020 (263h23m31.472142141s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.183707791 (263h23m32.614787922s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.694075084 (263h23m33.125155215s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.716788163 (263h23m34.147868294s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.851890717 (263h23m34.282970848s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:40.293306092 (263h23m34.724386243s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:42.156929229 (263h23m36.58800936s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157075889 (263h23m36.58815601s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157126075 (263h23m36.588206206s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection Aug 19 16:49:42.157170951 (263h23m36.588251082s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: read tcp 188.166.225.170:34242->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:42.157215747 (263h23m36.588295878s) [seg_pos={0 0}, upload=273839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 19 16:49:42.385478364 (263h23m36.816558495s) [upload=273839]: Done uploading segments... Aug 19 16:49:42.385503091 (263h23m36.816583212s) [upload=273839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.948030141s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [....................X.X...X...X..............X......X.XX....] duration: 6.035938866s FAIL (2025-08-19 16:52:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Aug 19 16:48:51.452491912 (263h22m45.883572043s) [upload=273834]: Starting upload Aug 19 16:48:51.452531028 (263h22m45.883611159s) [upload=273834]: Done starting upload Aug 19 16:48:51.452781006 (263h22m45.883861137s) [upload=273834]: Uploading segments... Aug 19 16:48:51.452853734 (263h22m45.883933866s) [upload=273834]: Got next segment. Inline: false Aug 19 16:48:51.452892670 (263h22m45.883972871s) [upload=273834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:48:51.452899813 (263h22m45.883980004s) [upload=273834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:48:51.962807465 (263h22m46.393887596s) [seg_pos={0 0}, upload=273834]: Begin upload segment... Aug 19 16:48:51.962852621 (263h22m46.393932743s) [seg_pos={0 0}, upload=273834]: Done begin upload segment. Aug 19 16:48:51.962875075 (263h22m46.393955206s) [noise=true , address=104.36.82.23:4470 , piece_id=5PDSSXU4HWGHBN62JJQBKHDQ6YYJVBWZZYSCJGKOPTZAZZUCAB6Q, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.962917746 (263h22m46.393997877s) [noise=true , address=66.23.193.175:4556 , piece_id=FBS7KZJF3HULYZZJDNLNOKMR5HSCR224IY7ZU4FUKZC5YMIIOAJA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963271022 (263h22m46.394351153s) [noise=true , address=45.91.116.9:4286 , piece_id=WYJJULBB3QVG2F46C52OCJWAMHVORGI7MDQS3CSYLF7REGAZQ3OQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963372556 (263h22m46.394452687s) [noise=true , address=156.146.43.225:4452 , piece_id=3T6HWTUPTX6A7BZULB4RSKGUO6GNDIHQGUP5W3EESJBWZGN7LLEQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963580684 (263h22m46.394660815s) [noise=true , address=104.36.82.35:4606 , piece_id=R37F22AIPFPV45K2PRWAJNONXQEDBJQPZGQKQ3LSHAQIBPKRXLHA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963653934 (263h22m46.394734066s) [noise=true , address=104.36.82.21:4478 , piece_id=XJWKXZ45KPRM3QUYXVIYU75M24GOCWGG5E3K4V62225XF3TGSS4Q, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963841874 (263h22m46.394922005s) [noise=true , address=79.127.201.215:4343 , piece_id=JF5HESY3ZARRZWZJHGTIG2BEZWWMXPHNJF5ALABQBAOVGJCHKI5A, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963910455 (263h22m46.394990586s) [noise=true , address=79.127.219.51:4211 , piece_id=OUB6QKVWCRJATMQFLAQ7UK3LJ4DILSLRAGFDP5YXROEXHZCPWGZA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964097293 (263h22m46.395177424s) [noise=true , address=104.36.82.60:4341 , piece_id=JCAHPCHE5IPHX35ZCDQ27I3DZ3R347HFRQWNBCNBXU6P3O7PTONA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964163640 (263h22m46.395243771s) [noise=true , address=77.90.31.18:4828 , piece_id=NFXT5LHIVX4G6DXMB4UWI37XCXLNHBRLVLXSJWTYAYN2JKCHRZQA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964400383 (263h22m46.395480514s) [noise=true , address=104.36.82.51:4052 , piece_id=7HLVSHJCDKI3TNKFETBUAY25HVIOPKAMSYQ6Q5BJ2NCJTQOTETTA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964624241 (263h22m46.395704372s) [noise=true , address=77.90.31.10:4529 , piece_id=HK5DLHGNULVF2RX7NNKFYN5JIFJB3LDEYXLXO3UN4D5OO4DJOW4A, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964832199 (263h22m46.39591233s) [noise=true , address=45.91.116.11:4130 , piece_id=5IZILPAZ4UKTFE766TWKT672JHDPWIQJI3TMOTASODK7NYG5BKPQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965048673 (263h22m46.396128804s) [noise=true , address=104.36.82.11:4127 , piece_id=6ACOOH24TTEMA4OQP6OFNHIABWDW3NDUVD44FSRIFLPTSVPHXOOA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965259626 (263h22m46.396339747s) [noise=true , address=77.90.31.15:4713 , piece_id=26S5ETN4OVNEL7IYYAIUBLP6UDS2QSSXCHD4H6YNHEQNY6FPOBOA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965452415 (263h22m46.396532546s) [noise=true , address=45.91.116.14:4209 , piece_id=5KAIVWUEWAG5NW6QGOAJLIMUADHZ6R4YTNAKTTXAGBRUEYXDAE7A, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965641637 (263h22m46.396721768s) [noise=true , address=45.91.116.7:4020 , piece_id=5NNTOPQ5BC7T2TLQ7NWSO3E6R4NDPB3JPXPMPRGG6I7YJU6QS4CQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965869794 (263h22m46.396949925s) [noise=true , address=77.90.31.17:4793 , piece_id=K3GCAOBIKBJV6Q3KOFXL5SEUFYE46MKJU5IAYNDA25IDCICTOFCQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:53.162668249 (263h22m47.59374838s) [upload=273834]: Next returned nil segment Aug 19 16:48:53.162683478 (263h22m47.593763599s) [upload=273834]: Waiting for error group managing segments... Aug 19 16:48:53.488132069 (263h22m47.91921223s) [noise=true , address=66.23.193.175:4556 , piece_id=FBS7KZJF3HULYZZJDNLNOKMR5HSCR224IY7ZU4FUKZC5YMIIOAJA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.887114541 (263h22m48.318194672s) [noise=true , address=79.127.219.51:4211 , piece_id=OUB6QKVWCRJATMQFLAQ7UK3LJ4DILSLRAGFDP5YXROEXHZCPWGZA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.925197211 (263h22m48.356277342s) [noise=true , address=156.146.43.225:4452 , piece_id=3T6HWTUPTX6A7BZULB4RSKGUO6GNDIHQGUP5W3EESJBWZGN7LLEQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.961937701 (263h22m48.393017832s) [noise=true , address=104.36.82.11:4127 , piece_id=6ACOOH24TTEMA4OQP6OFNHIABWDW3NDUVD44FSRIFLPTSVPHXOOA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.992310049 (263h22m48.42339018s) [noise=true , address=79.127.201.215:4343 , piece_id=JF5HESY3ZARRZWZJHGTIG2BEZWWMXPHNJF5ALABQBAOVGJCHKI5A, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.045243820 (263h22m48.476323961s) [noise=true , address=104.36.82.35:4606 , piece_id=R37F22AIPFPV45K2PRWAJNONXQEDBJQPZGQKQ3LSHAQIBPKRXLHA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.059875193 (263h22m48.490955324s) [noise=true , address=104.36.82.21:4478 , piece_id=XJWKXZ45KPRM3QUYXVIYU75M24GOCWGG5E3K4V62225XF3TGSS4Q, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.095629989 (263h22m48.52671012s) [noise=true , address=104.36.82.23:4470 , piece_id=5PDSSXU4HWGHBN62JJQBKHDQ6YYJVBWZZYSCJGKOPTZAZZUCAB6Q, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.103250568 (263h22m48.534330699s) [noise=true , address=104.36.82.51:4052 , piece_id=7HLVSHJCDKI3TNKFETBUAY25HVIOPKAMSYQ6Q5BJ2NCJTQOTETTA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.159963800 (263h22m48.591043921s) [noise=true , address=104.36.82.60:4341 , piece_id=JCAHPCHE5IPHX35ZCDQ27I3DZ3R347HFRQWNBCNBXU6P3O7PTONA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:49:37.214809078 (263h23m31.64588921s) [noise=true , address=77.90.31.17:4793 , piece_id=K3GCAOBIKBJV6Q3KOFXL5SEUFYE46MKJU5IAYNDA25IDCICTOFCQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:49:50.811468753 (263h23m45.242548894s) [noise=true , address=77.90.31.10:4529 , piece_id=HK5DLHGNULVF2RX7NNKFYN5JIFJB3LDEYXLXO3UN4D5OO4DJOW4A, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: read tcp 188.166.225.170:57426->77.90.31.10:4529: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811576489 (263h23m45.24265662s) [noise=true , address=45.91.116.14:4209 , piece_id=5KAIVWUEWAG5NW6QGOAJLIMUADHZ6R4YTNAKTTXAGBRUEYXDAE7A, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:12LWbAxsrDcyV31SxJNiVNYs7ERKAf6RTuKpE596UrYJAnXMndR, address:45.91.116.14:4209): protocol: rpc: dial tcp 45.91.116.14:4209: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811688974 (263h23m45.242769105s) [noise=true , address=45.91.116.9:4286 , piece_id=WYJJULBB3QVG2F46C52OCJWAMHVORGI7MDQS3CSYLF7REGAZQ3OQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:19DfbHig9tzF4VRHtSWkExwUS2bdZ8GFgKnAD9wvMBG7qaesNa, address:45.91.116.9:4286): protocol: read tcp 188.166.225.170:40032->45.91.116.9:4286: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811727057 (263h23m45.242807188s) [noise=true , address=45.91.116.11:4130 , piece_id=5IZILPAZ4UKTFE766TWKT672JHDPWIQJI3TMOTASODK7NYG5BKPQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:126ThYkfUAYf2JR8f5rkwyumcvmRArxKbpLUjhJukoc3FADHj2T, address:45.91.116.11:4130): protocol: read tcp 188.166.225.170:46860->45.91.116.11:4130: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811808934 (263h23m45.242889085s) [noise=true , address=45.91.116.7:4020 , piece_id=5NNTOPQ5BC7T2TLQ7NWSO3E6R4NDPB3JPXPMPRGG6I7YJU6QS4CQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:1MT5Nk9DRHLLfHi5sJnzWEjuL3aV2HnbS9Xebdw8vtBMjQmFnV, address:45.91.116.7:4020): protocol: read tcp 188.166.225.170:48966->45.91.116.7:4020: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811874559 (263h23m45.24295469s) [noise=true , address=77.90.31.18:4828 , piece_id=NFXT5LHIVX4G6DXMB4UWI37XCXLNHBRLVLXSJWTYAYN2JKCHRZQA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:1jwTDXPShXfNdfMx3UXi22gQphBpFre1MhRsHyjq6xRz4wwTr9, address:77.90.31.18:4828): protocol: read tcp 188.166.225.170:47612->77.90.31.18:4828: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811914556 (263h23m45.242994687s) [noise=true , address=77.90.31.15:4713 , piece_id=26S5ETN4OVNEL7IYYAIUBLP6UDS2QSSXCHD4H6YNHEQNY6FPOBOA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: read tcp 188.166.225.170:41506->77.90.31.15:4713: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811942720 (263h23m45.243022841s) [seg_pos={0 0}, upload=273834]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Aug 19 16:49:51.082762534 (263h23m45.513842665s) [upload=273834]: Done uploading segments... Aug 19 16:49:51.082792671 (263h23m45.513872803s) [upload=273834]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 6.035961518s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X...X.....] duration: 6.577131094s FAIL (2025-08-19 16:52:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.577151082s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................................X....XX...] duration: 6.518971933s FAIL (2025-08-19 16:52:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 19 16:49:12.801154313 (263h23m7.232234445s) [upload=273839]: Starting upload Aug 19 16:49:12.801215942 (263h23m7.232296073s) [upload=273839]: Done starting upload Aug 19 16:49:12.801330351 (263h23m7.232410482s) [upload=273839]: Uploading segments... Aug 19 16:49:12.801425372 (263h23m7.232505503s) [upload=273839]: Got next segment. Inline: false Aug 19 16:49:12.801455880 (263h23m7.232536012s) [upload=273839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:49:12.801461982 (263h23m7.232542113s) [upload=273839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:49:13.036708228 (263h23m7.467788359s) [seg_pos={0 0}, upload=273839]: Begin upload segment... Aug 19 16:49:13.036749197 (263h23m7.467829328s) [seg_pos={0 0}, upload=273839]: Done begin upload segment. Aug 19 16:49:13.036771971 (263h23m7.467852092s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.036780256 (263h23m7.467860437s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037135917 (263h23m7.468216048s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037326592 (263h23m7.468406723s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037403259 (263h23m7.46848339s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037667814 (263h23m7.468747945s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037761764 (263h23m7.468841896s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037943331 (263h23m7.469023462s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038070565 (263h23m7.469150696s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038168622 (263h23m7.469248753s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038300104 (263h23m7.469380235s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038396789 (263h23m7.46947692s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038622241 (263h23m7.469702362s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038679350 (263h23m7.469759481s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038854735 (263h23m7.469934856s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.039048045 (263h23m7.470128166s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:14.749064365 (263h23m9.180144496s) [upload=273839]: Next returned nil segment Aug 19 16:49:14.749117217 (263h23m9.180197348s) [upload=273839]: Waiting for error group managing segments... Aug 19 16:49:15.090317863 (263h23m9.521397994s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.132109007 (263h23m9.563189138s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.222090156 (263h23m9.653170287s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.328041780 (263h23m9.759121911s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.353998781 (263h23m9.785078912s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:36.709543777 (263h23m31.140623898s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:37.041062020 (263h23m31.472142141s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.183707791 (263h23m32.614787922s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.694075084 (263h23m33.125155215s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.716788163 (263h23m34.147868294s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.851890717 (263h23m34.282970848s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:40.293306092 (263h23m34.724386243s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:42.156929229 (263h23m36.58800936s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157075889 (263h23m36.58815601s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157126075 (263h23m36.588206206s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection Aug 19 16:49:42.157170951 (263h23m36.588251082s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: read tcp 188.166.225.170:34242->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:42.157215747 (263h23m36.588295878s) [seg_pos={0 0}, upload=273839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 19 16:49:42.385478364 (263h23m36.816558495s) [upload=273839]: Done uploading segments... Aug 19 16:49:42.385503091 (263h23m36.816583212s) [upload=273839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.519000117s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.814422479s FAIL (2025-08-19 16:52:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 263h18m33.75518017s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324344006 (263h18m33.755424127s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324633220 (263h18m33.755713351s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324851286 (263h18m33.755931417s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325068942 (263h18m33.756149063s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325283613 (263h18m33.756363744s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325499857 (263h18m33.756579998s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325759794 (263h18m33.756839915s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326059428 (263h18m33.757139549s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326277565 (263h18m33.757357696s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326470835 (263h18m33.757550966s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326673753 (263h18m33.757753884s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326867664 (263h18m33.757947795s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:40.851885389 (263h18m35.28296552s) [upload=273764]: Next returned nil segment Aug 19 16:44:40.851903334 (263h18m35.282983465s) [upload=273764]: Waiting for error group managing segments... Aug 19 16:44:41.244668372 (263h18m35.675748503s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.277280365 (263h18m35.708360496s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.328561665 (263h18m35.759641786s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.331747139 (263h18m35.76282727s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.357720320 (263h18m35.788800481s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.378190805 (263h18m35.809270936s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.486480333 (263h18m35.917560464s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:45:38.230570787 (263h19m32.661650918s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: expected piece hash; context deadline exceeded; EOF Aug 19 16:45:38.230979869 (263h19m32.66206s) [noise=true , address=77.90.31.18:4840 , piece_id=JWOETSFSZ3HELLZNLEASYNGW5OR64TMQVOFYYJAJVTHWCBHVTSLQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: read tcp 188.166.225.170:39946->77.90.31.18:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231017812 (263h19m32.662097943s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 188.166.225.170:44164->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231063489 (263h19m32.66214362s) [noise=true , address=45.91.116.14:4203 , piece_id=WRLBES3FI5EA5AQLZXJYGDROP3KBG3E2F22R4YD6A5KKI44PLV2Q, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 188.166.225.170:51738->45.91.116.14:4203: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231090952 (263h19m32.662171083s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: read tcp 188.166.225.170:57158->77.90.31.15:4707: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231110719 (263h19m32.66219085s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:50784->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231140165 (263h19m32.662220296s) [noise=true , address=77.90.31.16:4749 , piece_id=XCPXVBYLQH345FPIW4KDNLBFUCDDEW2H4B25WIXGOO5FVMAIVKPQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: read tcp 188.166.225.170:51668->77.90.31.16:4749: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231144814 (263h19m32.662224945s) [noise=true , address=77.90.31.19:4865 , piece_id=B3HUFIJZKVZJH4OBBO2DD7S3I7HX4LO2IQ74XJPO4DTW4QER35NA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1bguBghQonV5sTHaWteyUEnpp5KNzBKbvBpbvrUNFDCveHN77X, address:77.90.31.19:4865): protocol: read tcp 188.166.225.170:39398->77.90.31.19:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231177167 (263h19m32.662257298s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: read tcp 188.166.225.170:58526->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231184571 (263h19m32.662264702s) [noise=true , address=45.91.116.12:4236 , piece_id=LCRQS2GOSHOSLZBGQTGB3Q3PA4VN7KXNW66TDZ5O424TPJWTQE3A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH, address:45.91.116.12:4236): protocol: read tcp 188.166.225.170:57166->45.91.116.12:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231189770 (263h19m32.662269901s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 188.166.225.170:54242->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231215850 (263h19m32.662295981s) [seg_pos={0 0}, upload=273764]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 19 16:45:38.459100173 (263h19m32.890180304s) [upload=273764]: Done uploading segments... Aug 19 16:45:38.459128316 (263h19m32.890208447s) [upload=273764]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.814447238s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.................X...............X..........................] duration: 2.169841859s FAIL (2025-08-19 16:51:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after be5a5f4d20475bc6e6586ca4cb41bd260785c25704d0f801265da5a966efbd34 != before fab796bcbc8cefce4dfe0fedb7e23ecb2129705e75b1b66c720b4c58f0a82c58 ======LOGS====== Aug 19 16:31:34.462024854 (263h5m28.893104995s) [upload=273541]: Starting upload Aug 19 16:31:34.462073225 (263h5m28.893153356s) [upload=273541]: Done starting upload Aug 19 16:31:34.462124303 (263h5m28.893204434s) [upload=273541]: Uploading segments... Aug 19 16:31:34.462159490 (263h5m28.893239621s) [upload=273541]: Got next segment. Inline: true Aug 19 16:31:34.462163108 (263h5m28.893243239s) [upload=273541]: Waiting for error group managing segments... Aug 19 16:31:34.462196912 (263h5m28.893277044s) [upload=273541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:31:34.462200970 (263h5m28.893281101s) [upload=273541]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:31:34.462203415 (263h5m28.893283546s) [upload=273541]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:31:34.760143563 (263h5m29.191223694s) [upload=273541]: Done uploading segments... Aug 19 16:31:34.760161496 (263h5m29.191241628s) [upload=273541]: Upload finished. err: <nil> Duration:* 2.169893927s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....................X.X...X...X..............X......X.XX...] duration: 6.32427855s FAIL (2025-08-19 16:51:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Aug 19 16:48:51.452491912 (263h22m45.883572043s) [upload=273834]: Starting upload Aug 19 16:48:51.452531028 (263h22m45.883611159s) [upload=273834]: Done starting upload Aug 19 16:48:51.452781006 (263h22m45.883861137s) [upload=273834]: Uploading segments... Aug 19 16:48:51.452853734 (263h22m45.883933866s) [upload=273834]: Got next segment. Inline: false Aug 19 16:48:51.452892670 (263h22m45.883972871s) [upload=273834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:48:51.452899813 (263h22m45.883980004s) [upload=273834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:48:51.962807465 (263h22m46.393887596s) [seg_pos={0 0}, upload=273834]: Begin upload segment... Aug 19 16:48:51.962852621 (263h22m46.393932743s) [seg_pos={0 0}, upload=273834]: Done begin upload segment. Aug 19 16:48:51.962875075 (263h22m46.393955206s) [noise=true , address=104.36.82.23:4470 , piece_id=5PDSSXU4HWGHBN62JJQBKHDQ6YYJVBWZZYSCJGKOPTZAZZUCAB6Q, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.962917746 (263h22m46.393997877s) [noise=true , address=66.23.193.175:4556 , piece_id=FBS7KZJF3HULYZZJDNLNOKMR5HSCR224IY7ZU4FUKZC5YMIIOAJA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963271022 (263h22m46.394351153s) [noise=true , address=45.91.116.9:4286 , piece_id=WYJJULBB3QVG2F46C52OCJWAMHVORGI7MDQS3CSYLF7REGAZQ3OQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963372556 (263h22m46.394452687s) [noise=true , address=156.146.43.225:4452 , piece_id=3T6HWTUPTX6A7BZULB4RSKGUO6GNDIHQGUP5W3EESJBWZGN7LLEQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963580684 (263h22m46.394660815s) [noise=true , address=104.36.82.35:4606 , piece_id=R37F22AIPFPV45K2PRWAJNONXQEDBJQPZGQKQ3LSHAQIBPKRXLHA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963653934 (263h22m46.394734066s) [noise=true , address=104.36.82.21:4478 , piece_id=XJWKXZ45KPRM3QUYXVIYU75M24GOCWGG5E3K4V62225XF3TGSS4Q, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963841874 (263h22m46.394922005s) [noise=true , address=79.127.201.215:4343 , piece_id=JF5HESY3ZARRZWZJHGTIG2BEZWWMXPHNJF5ALABQBAOVGJCHKI5A, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963910455 (263h22m46.394990586s) [noise=true , address=79.127.219.51:4211 , piece_id=OUB6QKVWCRJATMQFLAQ7UK3LJ4DILSLRAGFDP5YXROEXHZCPWGZA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964097293 (263h22m46.395177424s) [noise=true , address=104.36.82.60:4341 , piece_id=JCAHPCHE5IPHX35ZCDQ27I3DZ3R347HFRQWNBCNBXU6P3O7PTONA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964163640 (263h22m46.395243771s) [noise=true , address=77.90.31.18:4828 , piece_id=NFXT5LHIVX4G6DXMB4UWI37XCXLNHBRLVLXSJWTYAYN2JKCHRZQA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964400383 (263h22m46.395480514s) [noise=true , address=104.36.82.51:4052 , piece_id=7HLVSHJCDKI3TNKFETBUAY25HVIOPKAMSYQ6Q5BJ2NCJTQOTETTA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964624241 (263h22m46.395704372s) [noise=true , address=77.90.31.10:4529 , piece_id=HK5DLHGNULVF2RX7NNKFYN5JIFJB3LDEYXLXO3UN4D5OO4DJOW4A, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964832199 (263h22m46.39591233s) [noise=true , address=45.91.116.11:4130 , piece_id=5IZILPAZ4UKTFE766TWKT672JHDPWIQJI3TMOTASODK7NYG5BKPQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965048673 (263h22m46.396128804s) [noise=true , address=104.36.82.11:4127 , piece_id=6ACOOH24TTEMA4OQP6OFNHIABWDW3NDUVD44FSRIFLPTSVPHXOOA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965259626 (263h22m46.396339747s) [noise=true , address=77.90.31.15:4713 , piece_id=26S5ETN4OVNEL7IYYAIUBLP6UDS2QSSXCHD4H6YNHEQNY6FPOBOA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965452415 (263h22m46.396532546s) [noise=true , address=45.91.116.14:4209 , piece_id=5KAIVWUEWAG5NW6QGOAJLIMUADHZ6R4YTNAKTTXAGBRUEYXDAE7A, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965641637 (263h22m46.396721768s) [noise=true , address=45.91.116.7:4020 , piece_id=5NNTOPQ5BC7T2TLQ7NWSO3E6R4NDPB3JPXPMPRGG6I7YJU6QS4CQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965869794 (263h22m46.396949925s) [noise=true , address=77.90.31.17:4793 , piece_id=K3GCAOBIKBJV6Q3KOFXL5SEUFYE46MKJU5IAYNDA25IDCICTOFCQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:53.162668249 (263h22m47.59374838s) [upload=273834]: Next returned nil segment Aug 19 16:48:53.162683478 (263h22m47.593763599s) [upload=273834]: Waiting for error group managing segments... Aug 19 16:48:53.488132069 (263h22m47.91921223s) [noise=true , address=66.23.193.175:4556 , piece_id=FBS7KZJF3HULYZZJDNLNOKMR5HSCR224IY7ZU4FUKZC5YMIIOAJA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.887114541 (263h22m48.318194672s) [noise=true , address=79.127.219.51:4211 , piece_id=OUB6QKVWCRJATMQFLAQ7UK3LJ4DILSLRAGFDP5YXROEXHZCPWGZA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.925197211 (263h22m48.356277342s) [noise=true , address=156.146.43.225:4452 , piece_id=3T6HWTUPTX6A7BZULB4RSKGUO6GNDIHQGUP5W3EESJBWZGN7LLEQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.961937701 (263h22m48.393017832s) [noise=true , address=104.36.82.11:4127 , piece_id=6ACOOH24TTEMA4OQP6OFNHIABWDW3NDUVD44FSRIFLPTSVPHXOOA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.992310049 (263h22m48.42339018s) [noise=true , address=79.127.201.215:4343 , piece_id=JF5HESY3ZARRZWZJHGTIG2BEZWWMXPHNJF5ALABQBAOVGJCHKI5A, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.045243820 (263h22m48.476323961s) [noise=true , address=104.36.82.35:4606 , piece_id=R37F22AIPFPV45K2PRWAJNONXQEDBJQPZGQKQ3LSHAQIBPKRXLHA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.059875193 (263h22m48.490955324s) [noise=true , address=104.36.82.21:4478 , piece_id=XJWKXZ45KPRM3QUYXVIYU75M24GOCWGG5E3K4V62225XF3TGSS4Q, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.095629989 (263h22m48.52671012s) [noise=true , address=104.36.82.23:4470 , piece_id=5PDSSXU4HWGHBN62JJQBKHDQ6YYJVBWZZYSCJGKOPTZAZZUCAB6Q, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.103250568 (263h22m48.534330699s) [noise=true , address=104.36.82.51:4052 , piece_id=7HLVSHJCDKI3TNKFETBUAY25HVIOPKAMSYQ6Q5BJ2NCJTQOTETTA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.159963800 (263h22m48.591043921s) [noise=true , address=104.36.82.60:4341 , piece_id=JCAHPCHE5IPHX35ZCDQ27I3DZ3R347HFRQWNBCNBXU6P3O7PTONA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:49:37.214809078 (263h23m31.64588921s) [noise=true , address=77.90.31.17:4793 , piece_id=K3GCAOBIKBJV6Q3KOFXL5SEUFYE46MKJU5IAYNDA25IDCICTOFCQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:49:50.811468753 (263h23m45.242548894s) [noise=true , address=77.90.31.10:4529 , piece_id=HK5DLHGNULVF2RX7NNKFYN5JIFJB3LDEYXLXO3UN4D5OO4DJOW4A, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: read tcp 188.166.225.170:57426->77.90.31.10:4529: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811576489 (263h23m45.24265662s) [noise=true , address=45.91.116.14:4209 , piece_id=5KAIVWUEWAG5NW6QGOAJLIMUADHZ6R4YTNAKTTXAGBRUEYXDAE7A, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:12LWbAxsrDcyV31SxJNiVNYs7ERKAf6RTuKpE596UrYJAnXMndR, address:45.91.116.14:4209): protocol: rpc: dial tcp 45.91.116.14:4209: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811688974 (263h23m45.242769105s) [noise=true , address=45.91.116.9:4286 , piece_id=WYJJULBB3QVG2F46C52OCJWAMHVORGI7MDQS3CSYLF7REGAZQ3OQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:19DfbHig9tzF4VRHtSWkExwUS2bdZ8GFgKnAD9wvMBG7qaesNa, address:45.91.116.9:4286): protocol: read tcp 188.166.225.170:40032->45.91.116.9:4286: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811727057 (263h23m45.242807188s) [noise=true , address=45.91.116.11:4130 , piece_id=5IZILPAZ4UKTFE766TWKT672JHDPWIQJI3TMOTASODK7NYG5BKPQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:126ThYkfUAYf2JR8f5rkwyumcvmRArxKbpLUjhJukoc3FADHj2T, address:45.91.116.11:4130): protocol: read tcp 188.166.225.170:46860->45.91.116.11:4130: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811808934 (263h23m45.242889085s) [noise=true , address=45.91.116.7:4020 , piece_id=5NNTOPQ5BC7T2TLQ7NWSO3E6R4NDPB3JPXPMPRGG6I7YJU6QS4CQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:1MT5Nk9DRHLLfHi5sJnzWEjuL3aV2HnbS9Xebdw8vtBMjQmFnV, address:45.91.116.7:4020): protocol: read tcp 188.166.225.170:48966->45.91.116.7:4020: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811874559 (263h23m45.24295469s) [noise=true , address=77.90.31.18:4828 , piece_id=NFXT5LHIVX4G6DXMB4UWI37XCXLNHBRLVLXSJWTYAYN2JKCHRZQA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:1jwTDXPShXfNdfMx3UXi22gQphBpFre1MhRsHyjq6xRz4wwTr9, address:77.90.31.18:4828): protocol: read tcp 188.166.225.170:47612->77.90.31.18:4828: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811914556 (263h23m45.242994687s) [noise=true , address=77.90.31.15:4713 , piece_id=26S5ETN4OVNEL7IYYAIUBLP6UDS2QSSXCHD4H6YNHEQNY6FPOBOA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: read tcp 188.166.225.170:41506->77.90.31.15:4713: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811942720 (263h23m45.243022841s) [seg_pos={0 0}, upload=273834]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Aug 19 16:49:51.082762534 (263h23m45.513842665s) [upload=273834]: Done uploading segments... Aug 19 16:49:51.082792671 (263h23m45.513872803s) [upload=273834]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 6.324305293s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X....X...] duration: 6.091355298s FAIL (2025-08-19 16:51:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.091379163s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X...X....] duration: 6.707268125s FAIL (2025-08-19 16:51:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.707285408s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)